Patent Grant
7979654
The present invention relates generally to continuous data protection, and more particularly, to restoring a volume in a continuous data protection system.
Hardware redundancy schemes have traditionally been used in enterprise environments to protect against component failures. Redundant arrays of independent disks (RAID) have been implemented successfully to assure continued access to data even in the event of one or more media failures (depending on the RAID Level). Unfortunately, hardware redundancy schemes are ineffective in dealing with logical data loss or corruption. For example, an accidental file deletion or virus infection is automatically replicated to all of the redundant hardware components and can neither be prevented nor recovered from by such technologies. To overcome this problem, backup technologies have traditionally been deployed to retain multiple versions of a production system over time. This allowed administrators to restore previous versions of data and to recover from data corruption.
Backup copies are generally policy-based, are tied to a periodic schedule, and reflect the state of a primary volume (i.e., a protected volume) at the particular point in time that is captured. Because backups are not made on a continuous basis, there will be some data loss during the restoration, resulting from a gap between the time when the backup was performed and the restore point that is required. This gap can be significant in typical environments where backups are only performed once per day. In a mission-critical setting, such a data loss can be catastrophic. Beyond the potential data loss, restoring a primary volume from a backup system can be complicated and often takes many hours to complete. This additional downtime further exacerbates the problems associated with a logical data loss.
The traditional process of backing up data to tape media is time driven and time dependent. That is, a backup process typically is run at regular intervals and covers a certain period of time. For example, a full system backup may be run once a week on a weekend, and incremental backups may be run every weekday during an overnight backup window that starts after the close of business and ends before the next business day. These individual backups are then saved for a predetermined period of time, according to a retention policy. In order to conserve tape media and storage space, older backups are gradually faded out and replaced by newer backups. Further to the above example, after a full weekly backup is completed, the daily incremental backups for the preceding week may be discarded, and each weekly backup may be maintained for a few months, to be replaced by monthly backups. The daily backups are typically not all discarded on the same day. Instead, the Monday backup set is overwritten on Monday, the Tuesday backup set is overwritten on Tuesday, and so on. This ensures that a backup set is available that is within eight business hours of any corruption that may have occurred in the past week.
Despite frequent hardware failures and the necessity of ongoing maintenance and tuning, the backup creation process can be automated, while restoring data from a backup remains a manual and time-critical process. First, the appropriate backup tapes need to be located, including the latest full backup and any incremental backups made since the last full backup. In the event that only a partial restoration is required, locating the appropriate backup tape can take just as long. Once the backup tapes are located, they must be restored to the primary volume. Even under the best of circumstances, this type of backup and restore process cannot guarantee high availability of data.
Another type of data protection involves making point in time (PIT) copies of data. A first type of PIT copy is a hardware-based PIT copy, which is a mirror of the primary volume onto a secondary volume. The main drawbacks to a hardware-based PIT copy are that the data ages quickly and that each copy takes up as much disk space as the primary volume. A software-based PIT, typically called a “snapshot,” is a “picture” of a volume at the block level or a file system at the operating system level. Various types of software-based PITs exist, and most are tied to a particular platform, operating system, or file system. These snapshots also have drawbacks, including occupying additional space on the primary volume, rapid aging, and possible dependencies on data stored on the primary volume wherein data corruption on the primary volume leads to corruption of the snapshot. In addition, snapshot systems generally do not offer the flexibility in scheduling and expiring snapshots that backup software provides.
While both hardware-based and software-based PIT techniques reduce the dependency on the backup window, they still require the traditional tape-based backup and restore process to move data from disk to tape media and to manage the different versions of data. This dependency on legacy backup applications and processes is a significant drawback of these technologies. Furthermore, like traditional tape-based backup and restore processes, PIT copies are made at discrete moments in time, thereby limiting any restores that are performed to the points in time at which PIT copies have been made.
A need therefore exists for a system that combines the advantages of tape-based systems with the advantages of snapshot systems and eliminates the limitations described above.
A method for synchronizing a secondary volume with a primary volume in a continuous data protection system begins by scanning a region of the primary volume. The scanned region is then compared with a corresponding region of the secondary volume. An identification of the scanned region is stored in a compare delta map when the comparison results in a discrepancy between the scanned region and the corresponding region. Data is copied from the primary volume to the secondary volume, using the compare delta map as a guide to locate the data to copy. If the data protection system has failed just prior to the restore process being initiated, it is termed a re-baseline, and the entire primary volume is scanned. If the data protection system is active prior to the restore process being initiated, it is termed a re-synchronization, and is optimized by scanning only select regions of the primary volume. A dirty region log is used to maintain a list of those regions of the primary volume that are to be scanned.
A method for restoring a primary volume from a secondary volume in a continuous data protection system includes the steps of selecting a snapshot of the primary volume to be restored and loading the snapshot from the secondary volume to the primary volume.
A system for synchronizing a secondary volume with a primary volume in a continuous data protection system includes scanning means for scanning a region of the primary volume; comparing means for comparing the scanned region with a corresponding region of the secondary volume; storing means for storing an identification of the scanned region in a compare delta map when the comparing means returns a discrepancy between the scanned region and the corresponding region; copying means for copying data from the primary volume to the secondary volume, using the compare delta map as a guide to locate the data to copy and revising means for revising the compare delta map by removing any changes made to the primary volume during a scan interval, which is a period of time required to scan the primary volume. If the data protection system has failed just prior to the restore process being initiated, it is termed a re-baseline, and the entire primary volume is scanned. If the data protection system is active prior to the restore process being initiated, it is termed a re-synchronization, and is optimized by scanning only select regions of the primary volume. A dirty region log is used to maintain a list of those regions of the primary volume that are to be scanned.
A system for restoring a primary volume from a secondary volume in a continuous data protection system includes at least one snapshot of the primary volume, each snapshot corresponding to a different point in time; selecting means for selecting a snapshot to be restored; and loading means for loading the selected snapshot from the secondary volume to the primary volume.
A more detailed understanding of the invention may be had from the following description of a preferred embodiment, given by way of example, and to be understood in conjunction with the accompanying drawings, wherein:
In the present invention, data is backed up continuously, allowing system administrators to pause, rewind, and replay live enterprise data streams. This moves the traditional backup methodologies into a continuous background process in which policies automatically manage the lifecycle of many generations of restore images.
System Construction
A volume manager is a software module that runs on a server or intelligent storage switch to manage storage resources. Typical volume managers have the ability to aggregate blocks from multiple different physical disks into one or more virtual volumes. Applications are not aware that they are actually writing to segments of many different disks because they are presented with one large, contiguous volume. In addition to block aggregation, volume managers usually also offer software RAID functionality. For example, they are able to split the segments of the different volumes into two groups, where one group is a mirror of the other group. This is, in a preferred embodiment, the feature that the data protection system is taking advantage of when the present invention is implemented as shown in
It is noted that the primary data volume 104 and the secondary data volume 108 can be any type of data storage, including, but not limited to, a single disk, a disk array (such as a RAID), or a storage area network (SAN). The main difference between the primary data volume 104 and the secondary data volume 108 lies in the structure of the data stored at each location, as will be explained in detail below. It is noted that there may also be differences in terms of the technologies that are used. The primary volume 104 is typically an expensive, fast, and highly available storage subsystem, whereas the secondary volume 108 is typically cost-effective, high capacity, and comparatively slow (for example, ATA/SATA disks). Normally, the slower secondary volume cannot be used as a synchronous mirror to the high-performance primary volume, because the slower response time will have an adverse impact on the overall system performance.
The data protection system 106, however, is optimized to keep up with high-performance primary volumes. These optimizations are described in more detail below, but at a high level, random writes to the primary volume 104 are processed sequentially on the secondary volume 108. Sequential writes improve both the cache behavior and the actual volume performance of the secondary volume 108. In addition, it is possible to aggregate multiple sequential writes on the secondary volume 108, whereas this is not possible with the random writes to the primary volume 104. The present invention does not require writes to the data protection system 106 to be synchronous. However, even in the case of an asynchronous mirror, minimizing latencies is important.
It is noted that the data protection system 106 operates in the same manner, regardless of the particular construction of the protected computer system 100, 120, 140. The major difference between these deployment options is the manner and place in which a copy of each write is obtained. To those skilled in the art it is evident that other embodiments, such as the cooperation between a switch platform and an external server, are also feasible.
Conceptual Overview
To facilitate further discussion, it is necessary to explain some fundamental concepts associated with a continuous data protection system constructed in accordance with the present invention. In practice, certain applications require continuous data protection with a block-by-block granularity, for example, to rewind individual transactions. However, the period in which such fine granularity is required is generally short (for example, two days), which is why the system can be configured to fade out data over time. The present invention discloses data structures and methods to manage this process automatically.
The present invention keeps a log of every write made to a primary volume (a “write log”) by duplicating each write and directing the copy to a cost-effective secondary volume in a sequential fashion. The resulting write log on the secondary volume can then be played back one write at a time to recover the state of the primary volume at any previous point in time. Replaying the write log one write at a time is very time consuming, particularly if a large amount of write activity has occurred since the creation of the write log. In typical recovery scenarios, it is necessary to examine how the primary volume looked like at multiple points in time before deciding which point to recover to. For example, consider a system that was infected by a virus. In order to recover from the virus, it is necessary to examine the primary volume as it was at different points in time to find the latest recovery point where the system was not yet infected by the virus. Additional data structures are needed to efficiently compare multiple potential recovery points.
Delta Maps
Delta maps provide a mechanism to efficiently recover the primary volume as it was at a particular point in time without the need to replay the write log in its entirety, one write at a time. In particular, delta maps are data structures that keep track of data changes between two points in time. These data structures can then be used to selectively play back portions of the write log such that the resulting point-in-time image is the same as if the log were played back one write at a time, starting at the beginning of the log.
As shown in delta map 200, row 210 is an originating entry and row 220 is a terminating entry. Row 210 includes a field 212 for specifying the region of a primary volume where the first block was written, a field 214 for specifying the block offset in the region of the primary volume where the write begins, a field 216 for specifying where on the secondary volume the duplicate write (i.e., the copy of the primary volume write) begins, and a field 218 for specifying the physical device (the physical volume or disk identification) used to initiate the write. Row 220 includes a field 222 for specifying the region of the primary volume where the last block was written, a field 224 for specifying the block offset in the region of the primary volume where the write ends, a field 226 for specifying the where on the secondary volume the duplicate write ends, and a field 228. While fields 226 and 228 are provided in a terminating entry such as row 220, it is noted that field 226 is optional because this value can be calculated by subtracting the offsets of the originating entry and the terminating entry (field 226=(field 224−field 214)+field 216), and field 228 is not necessary since there is no physical device usage associated with termination of a write.
In a preferred embodiment, as explained above, each delta map contains a list of all blocks that were changed during the particular time period to which the delta map corresponds. That is, each delta map specifies a block region on the primary volume, the offset on the primary volume, and physical device information. It is noted, however, that other fields or a completely different mapping format may be used while still achieving the same functionality. For example, instead of dividing the primary volume into block regions, a bitmap could be kept, representing every block on the primary volume. Once the retention policy (which is set purely according to operator preference) no longer requires the restore granularity to include a certain time period, corresponding blocks are freed up, with the exception of any blocks that may still be necessary to restore to later recovery points. Once a particular delta map expires, its block list is returned to the appropriate block allocator for re-use.
Delta maps are initially created from the write log using a map engine, and can be created in real-time, after a certain number of writes, or according to a time interval. It is noted that these are examples of ways to trigger the creation of a delta map, and that one skilled in the art could devise various other triggers. Additional delta maps may also be created as a result of a merge process (called “merged delta maps”) and may be created to optimize the access and restore process. The delta maps are stored on the secondary volume and contain a mapping of the primary address space to the secondary address space. The mapping is kept in sorted order based on the primary address space.
One significant benefit of merging delta maps is a reduction in the number of delta map entries that are required. For example, when there are two writes that are adjacent to each other on the primary volume, the terminating entry for the first write can be eliminated from the merged delta map, since its location is the same as the originating entry for the second write. The delta maps and the structures created by merging maps reduces the amount of overhead required in maintaining the mapping between the primary and secondary volumes.
Dirty Region Log
The purpose of the dirty region log (DRL) mechanism is to track write operations that are in memory but not logged to the secondary volume. Depending on the particular deployment used (shown in
The DRL is implemented in a preferred embodiment as a bit map with the bits representing equally sized regions of the primary volume address space. Each region typically contains multiple logical blocks of the primary volume. A bit set in the DRL indicates that a write operation has been (or at least may have been) initiated for the corresponding region of the primary volume. Since the completion status of the write operation is unknown, that region of the primary volume must be considered as potentially changed or “dirty.”
For an example of the efficiency gained by using the DRL, assume that the primary volume is 100 GB and is divided into 1 GB blocks. Further assume that only one of the blocks is marked as dirty. If no DRL was used, the entire 100 GB of the primary volume would need to be compared with the secondary volume in order to resynchronize the two volumes. At a typical sequential access speed of 50 MB per second for a single disk, reading 100 GB of data on the primary volume would take over 33 minutes. In comparison, since the DRL indicates that only one block needs to be read, the read operation would only take 20 seconds. The need for such an optimization is more readily apparent when dealing with larger disk sizes typically encountered in an enterprise environment. As a second example, with 10 TB of primary storage, synchronizing the primary volume could take over two days (at a read speed of 50 MB per second) if no DRL is used.
During normal operation, write input/outputs (I/Os) are first tested against the current DRL to see if the region of the primary volume affected by the write operation is already “dirty.” If it is, the write I/O is then immediately entered into the current write log and the write operations to the primary and secondary volumes are initiated. If the bit in the DRL corresponding to the write operation is clear, indicating that the region on the primary volume is “clean,” then the bit is set (indicating a change to the primary data in that region) and the DRL is made persistent before the write operation is acknowledged. However, in an asynchronous implementation, the acknowledgement can occur before the secondary write is entered into the current write log. When the secondary volume acknowledges the write, the bit in the DRL is cleared.
The synchronous nature of the DRL change and commit is important. It is permissible for the DRL to have bits set even if data changes did not occur to the primary volume (e.g., the write I/O fails or is not issued due to a software failure), but it is not permissible for the DRL to have bits which are clear if there is a chance that the corresponding regions on the primary volume may have been changed.
Periodically as determined by policy, the DRL is reset by clearing all the bits in the map except for the bits representing regions of the primary volume for which uncommitted write I/Os exist (e.g., in write logs in memory but not yet written to disk). During certain critical phases of the re-synchronization workflow, it is important that the DRL be prevented from resetting. This is accomplished indirectly by manipulating the write log. The DRL reset policy will not allow the DRL to be reset when either the write log is not active or the write log is active but currently “Out of Sync.” Deactivating the write log or inserting an “Out of Sync” marker into an active write log effectively disables DRL resets, while inserting an “In Sync” marker into an active write log effectively enables them.
During re-synchronization, regions on the primary volume corresponding to “dirty” bits in the DRL are compared to secondary volume data as mapped by the most recent reliable PIT map. The re-synchronization function can then bring the secondary volume data into agreement with the primary volume data if they are found to be different.
It is noted that there is a natural tension between the latency introduced by synchronous DRL writes (required when new regions of the primary volume are “dirtied”) and the time required to re-synchronize the data after a failure. Over time, as the DRL fills with “dirty” bits, the fewer synchronous writes of the DRL are required, reducing the average I/O latency. However, the more filled the DRL becomes with “dirty” bits, the more regions of the primary volume must be scanned during re-synchronization. Resetting the DRL too frequently will reduce the time needed to re-synchronize at the expense of increased latency, while too few DRL resets will reduce latency at the expense of an overly long recovery time. The trade-off between latency and recovery time is handled by the DRL reset policy.
Re-Baseline
The re-baseline procedure is used to fill in the gaps where the data protection system, including the DRL, has failed. In general, the re-baseline procedure operates similar to the creation of an initial snapshot, with the main difference being that only a fraction of the data has changed. Because certain optimizations exist, it is only necessary to fill in the gap in time where the failure occurred.
Over time, it is possible (although unlikely) that errors will accumulate within the data protection system with the result that the data on the primary and secondary volumes will no longer be in sync. The re-baseline procedure is a mechanism for detecting discrepancies between the primary and secondary volumes, marking suspect point-in-time intervals, and bringing the volumes back into synchronization. The re-baseline procedure can be run at any time to check the integrity of the primary and secondary volumes, and can fix any problems found in the background.
In general terms, the re-baseline procedure scans the entire primary and secondary volumes for data miscompares. When they are found, errors are corrected by copying data from the primary volume to the secondary volume and creating a “fix-up” delta map placed at the end of the delta map chain being maintained for the primary volume. The fix-up delta map brings the two volumes back into synchronization and restores any point in time (APIT) data protection. From this point in the map chain, the re-baseline procedure searches back through the map chain until it arrives at a map before which there are no errors. The interval (delta map sub-chain) between the earliest error-free map and the fix-up map is marked as being not reliable.
The re-baseline procedure introduces the concept of a “suspect” delta map, which is the system's way of delineating an interval of maps in the delta map chain through which errors have propagated. The suspect interval begins with the first map for which errors have been detected and continues forward in time until the fix-up delta map is reached. Each map within the interval is marked as suspect (except for the fix-up map), whether it contains mappings to regions with errors or not.
The significance of a suspect map is that if it is the most recent map in a map merge operation, the resulting merged map is also suspect. A suspect map merged with a more recent fix-up map results in a clean (non-suspect) map. A suspect map may only be merged with a more recent suspect map or a more recent fix-up map, i.e., it is by definition impossible to have a more recent non-suspect map other than a fix-up map adjacent to a suspect map in the map chain. While merging two suspect maps would result in a suspect merged map, it is still desirable to create the most recent image of the delta map even in the suspect time period. The reason for this is that if a user attempts to recover the volume to a time within this period, the user will be presented with a warning, but will still be permitted to examine the volume. It is possible that the specific blocks or files the user is looking for have not been corrupted. The map manager uses the suspect map marker to keep track of how reliable various composite maps are when they are built from the fundamental delta map chain elements.
The re-baseline procedure 300 begins (step 302; point B in
A miscompare delta map (Δmc) is created to track any miscompares (step 308). A region of the primary volume is scanned (step 310) and the scanned region is compared with the corresponding region of the secondary volume as indicated by the PIT map (step 312). This part of the process (steps 310 and 312) can be optimized in some situations by utilizing the DRL. Care must be taken when dealing with suspect volumes, because the DRL may not provide accurate information, and therefore cannot be used. If the DRL is current, then the re-synchronization procedure can be used, as discussed below. The safest method is to scan the entire volume, especially since the scanning can be performed in the background while the system is processing other requests. It is possible to use checkpoints such that one region of the volumes can be scanned and compared at a time, and if the system fails for some reason during the scan and compare steps, it will not be necessary to restart the entire re-baseline procedure from the beginning.
A determination is made whether the regions on the primary and secondary volumes match (step 314). If the regions do not match, this indicates that there is a miscompare between the primary and secondary volumes, and an entry is made into the miscompare delta map (step 316). If the regions match (step 314), then there are no miscompares between the two regions. Regardless of the outcome of step 314, another decision is made whether all of the regions of the primary and secondary volumes have been compared (step 318). As noted above, the DRL may be used to optimize this decision by only looking at the regions marked as “dirty” in the DRL. If all of the regions on the primary volume have not been compared, then the procedure continues at step 310.
If all of the regions on the primary volume have been compared (step 318), then a scan delta map (Δscan) is created for the scan interval (between points B and C in
A copy delta map (Δcopy) is created to store only the true miscompares between the primary and secondary volumes (step 322). In order to bring the primary and secondary volumes into agreement, data must be copied from the primary volume to the secondary volume for those areas where true miscompares have been identified. The copy delta map is used to guide the copy process by factoring out any changes made to the primary volume during the scan interval that may have resulted in false miscompares. This map is constructed by subtracting from the miscompare map (Δmc) the intersection of the miscompare map (Δmc) and the scan interval map (Δscan), as shown in Equation 1.
Δcopy=Δmc−(Δmc∩Δscan) Equation (1)
A determination is made whether the copy delta map is empty (step 324). If the copy delta map is empty, indicating that there are no true miscompares between the primary and secondary volumes (i.e., that the primary and secondary volumes are synchronized), then the procedure terminates (step 326).
If the copy delta map is not empty (step 324), then data from the primary volume is copied onto the secondary volume for those areas where miscompares were detected, overwriting the areas on the secondary volume, using the copy delta map and the current PIT map (step 328; between points C and D in
A fix-up delta map (Δfix-up) is created by merging the copy delta map into the host change delta map (step 332), as shown in Equation 2.
Δfix-up=Δhost←Δcopy Equation (2)
The fix-up delta map is then added to the end of the delta map chain (step 334). The fix-up delta map is used to bring the primary and secondary volumes into synchronization since it includes all of the changes made to the primary volume from the beginning of the scan interval (point B in
Because the copy delta map is not empty, errors existed in the delta map chain prior to the beginning of the scan interval (point B in
Δcopy=Δcopy−(Δcopy∩Δselected) Equation (3)
A determination is made whether the adjusted copy delta map is now empty (step 342). If the copy delta map is empty, indicating that there are no additional suspect delta maps to locate, then the procedure terminates (step 326). If the copy delta map is not empty (step 342), then the previous delta map in the map chain is selected (step 344) and the procedure continues with step 338. The point in the delta map chain preceding the earliest delta map to be marked as suspect (point A in
Re-Synchronization
If the DRL is correct (meaning that the data protection system has not failed) and the secondary volume fails, then a re-synchronization procedure can be used to update the secondary volume. The re-synchronization procedure operates in the same manner as the re-baseline procedure described above in connection with
Full Restore
The full restore procedure is used to place a snapshot onto the primary volume, in the event of a primary volume failure. Data is copied from the snapshot into the primary volume. It is possible to roll forward changes in the data from the time of the snapshot to the present time, at the user's preference. With appropriate file system decoders, it is even possible to selectively apply changes to certain data (i.e., a user can roll forward changes on a particular file), so the full restore procedure may function like a concurrent version system, as is used in software development.
The idea of a full restore is that once a user has selected a particular snapshot, the corresponding content (i.e., the new volume) is moved back to the primary volume, overwriting the corrupted contents there. If a host-based agent or service on a storage switch is used, requests to regions that have already been copied back to the primary volume can be served from there (and thus faster), whereas requests for blocks that have not been copied are served from the secondary volume. This makes the restore process invisible to the user, with the only noticeable difference being that initial requests will be a bit slower than usual.
While specific embodiments of the present invention have been shown and described, many modifications and variations could be made by one skilled in the art without departing from the scope of the invention. The above description serves to illustrate and not limit the particular invention in any way.
This application is a continuation of U.S. patent application Ser. No. 10/772,642, filed Feb. 5, 2004, now U.S. Pat. No. 7,426,617 which claims the benefit of U.S. Provisional Application No. 60/541,626 filed Feb. 4, 2004 and U.S. Provisional Application No. 60/542,011 filed Feb. 5, 2004, which are incorporated by reference as if fully set forth herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4635145 | Horie et al. | Jan 1987 | A |
| 4727512 | Birkner et al. | Feb 1988 | A |
| 4775969 | Osterlund | Oct 1988 | A |
| 5163148 | Walls | Nov 1992 | A |
| 5235695 | Pence | Aug 1993 | A |
| 5269022 | Shinjo et al. | Dec 1993 | A |
| 5297124 | Plotkin et al. | Mar 1994 | A |
| 5438674 | Keele et al. | Aug 1995 | A |
| 5455926 | Keele et al. | Oct 1995 | A |
| 5485321 | Leonhardt et al. | Jan 1996 | A |
| 5555371 | Duyanovich et al. | Sep 1996 | A |
| 5638509 | Dunphy et al. | Jun 1997 | A |
| 5666538 | DeNicola | Sep 1997 | A |
| 5673382 | Cannon et al. | Sep 1997 | A |
| 5774292 | Georgiou et al. | Jun 1998 | A |
| 5774643 | Lubbers et al. | Jun 1998 | A |
| 5774715 | Madany et al. | Jun 1998 | A |
| 5805864 | Carlson et al. | Sep 1998 | A |
| 5809511 | Peake | Sep 1998 | A |
| 5809543 | Byers et al. | Sep 1998 | A |
| 5835953 | Ohran | Nov 1998 | A |
| 5854720 | Shrinkle et al. | Dec 1998 | A |
| 5857208 | Ofek | Jan 1999 | A |
| 5864346 | Yokoi et al. | Jan 1999 | A |
| 5872669 | Morehouse et al. | Feb 1999 | A |
| 5875479 | Blount et al. | Feb 1999 | A |
| 5911779 | Stallmo et al. | Jun 1999 | A |
| 5949970 | Sipple et al. | Sep 1999 | A |
| 5961613 | DeNicola | Oct 1999 | A |
| 5963971 | Fosler et al. | Oct 1999 | A |
| 5974424 | Schmuck et al. | Oct 1999 | A |
| 6021408 | Ledain et al. | Feb 2000 | A |
| 6021491 | Renaud | Feb 2000 | A |
| 6023709 | Anglin et al. | Feb 2000 | A |
| 6029179 | Kishi | Feb 2000 | A |
| 6041329 | Kishi | Mar 2000 | A |
| 6044442 | Jesionowski | Mar 2000 | A |
| 6049848 | Yates et al. | Apr 2000 | A |
| 6061309 | Gallo et al. | May 2000 | A |
| 6067587 | Miller et al. | May 2000 | A |
| 6070224 | LeCrone et al. | May 2000 | A |
| 6098148 | Carlson | Aug 2000 | A |
| 6128698 | Georgis | Oct 2000 | A |
| 6131142 | Kamo et al. | Oct 2000 | A |
| 6131148 | West et al. | Oct 2000 | A |
| 6134660 | Boneh et al. | Oct 2000 | A |
| 6163856 | Dion et al. | Dec 2000 | A |
| 6173293 | Thekkath et al. | Jan 2001 | B1 |
| 6173359 | Carlson et al. | Jan 2001 | B1 |
| 6195730 | West | Feb 2001 | B1 |
| 6225709 | Nakajima | May 2001 | B1 |
| 6247096 | Fisher et al. | Jun 2001 | B1 |
| 6260110 | LeCrone et al. | Jul 2001 | B1 |
| 6266784 | Hsiao et al. | Jul 2001 | B1 |
| 6269423 | Kishi | Jul 2001 | B1 |
| 6269431 | Dunham | Jul 2001 | B1 |
| 6282609 | Carlson | Aug 2001 | B1 |
| 6289425 | Blendermann et al. | Sep 2001 | B1 |
| 6292889 | Fitzgerald et al. | Sep 2001 | B1 |
| 6301677 | Squibb | Oct 2001 | B1 |
| 6304880 | Kishi | Oct 2001 | B1 |
| 6317814 | Blendermann et al. | Nov 2001 | B1 |
| 6324497 | Yates et al. | Nov 2001 | B1 |
| 6327418 | Barton | Dec 2001 | B1 |
| 6336163 | Brewer et al. | Jan 2002 | B1 |
| 6336173 | Day et al. | Jan 2002 | B1 |
| 6339778 | Kishi | Jan 2002 | B1 |
| 6341329 | LeCrone et al. | Jan 2002 | B1 |
| 6343342 | Carlson | Jan 2002 | B1 |
| 6353837 | Blumenau | Mar 2002 | B1 |
| 6360232 | Brewer et al. | Mar 2002 | B1 |
| 6389503 | Georgis et al. | May 2002 | B1 |
| 6397307 | Ohran | May 2002 | B2 |
| 6408359 | Ito et al. | Jun 2002 | B1 |
| 6487561 | Ofek et al. | Nov 2002 | B1 |
| 6496791 | Yates et al. | Dec 2002 | B1 |
| 6499026 | Rivette et al. | Dec 2002 | B1 |
| 6557073 | Fujiwara et al. | Apr 2003 | B1 |
| 6557089 | Reed et al. | Apr 2003 | B1 |
| 6578120 | Crockett et al. | Jun 2003 | B1 |
| 6615365 | Jenevein et al. | Sep 2003 | B1 |
| 6625704 | Winokur | Sep 2003 | B2 |
| 6654912 | Viswanathan et al. | Nov 2003 | B1 |
| 6658435 | McCall | Dec 2003 | B1 |
| 6694447 | Leach et al. | Feb 2004 | B1 |
| 6725331 | Kedem | Apr 2004 | B1 |
| 6766520 | Rieschl et al. | Jul 2004 | B1 |
| 6779057 | Masters et al. | Aug 2004 | B2 |
| 6779058 | Kishi et al. | Aug 2004 | B2 |
| 6779081 | Arakawa et al. | Aug 2004 | B2 |
| 6796489 | Slater et al. | Sep 2004 | B2 |
| 6816941 | Carlson et al. | Nov 2004 | B1 |
| 6816942 | Okada et al. | Nov 2004 | B2 |
| 6834324 | Wood | Dec 2004 | B1 |
| 6839843 | Bacha et al. | Jan 2005 | B1 |
| 6850964 | Brough et al. | Feb 2005 | B1 |
| 6877016 | Hart et al. | Apr 2005 | B1 |
| 6898600 | Fruchtman et al. | May 2005 | B2 |
| 6915397 | Lubbers et al. | Jul 2005 | B2 |
| 6931557 | Togawa | Aug 2005 | B2 |
| 6950263 | Suzuki et al. | Sep 2005 | B2 |
| 6973369 | Trimmer et al. | Dec 2005 | B2 |
| 6973534 | Dawson | Dec 2005 | B2 |
| 6978283 | Edwards et al. | Dec 2005 | B1 |
| 6978325 | Gibble et al. | Dec 2005 | B2 |
| 7007043 | Farmer et al. | Feb 2006 | B2 |
| 7020779 | Sutherland | Mar 2006 | B1 |
| 7032126 | Zalewski et al. | Apr 2006 | B2 |
| 7055009 | Factor et al. | May 2006 | B2 |
| 7072910 | Kahn et al. | Jul 2006 | B2 |
| 7096331 | Haase et al. | Aug 2006 | B1 |
| 7100089 | Phelps | Aug 2006 | B1 |
| 7111136 | Yamagami | Sep 2006 | B2 |
| 7111194 | Schoenthal et al. | Sep 2006 | B1 |
| 7127388 | Yates et al. | Oct 2006 | B2 |
| 7127577 | Koning et al. | Oct 2006 | B2 |
| 7139891 | Apvrille et al. | Nov 2006 | B1 |
| 7152077 | Veitch et al. | Dec 2006 | B2 |
| 7152078 | Yamagami | Dec 2006 | B2 |
| 7155465 | Lee et al. | Dec 2006 | B2 |
| 7155586 | Wagner et al. | Dec 2006 | B1 |
| 7200726 | Gole et al. | Apr 2007 | B1 |
| 7203726 | Hasegawa | Apr 2007 | B2 |
| 7251713 | Zhang | Jul 2007 | B1 |
| 7302057 | Rotholtz et al. | Nov 2007 | B2 |
| 7346623 | Prahlad et al. | Mar 2008 | B2 |
| 7373364 | Chapman | May 2008 | B1 |
| 7558839 | McGovern | Jul 2009 | B1 |
| 7774610 | McGovern et al. | Aug 2010 | B2 |
| 7904679 | Stager et al. | Mar 2011 | B2 |
| 20010047447 | Katsuda | Nov 2001 | A1 |
| 20020004835 | Yarbrough | Jan 2002 | A1 |
| 20020016827 | McCabe et al. | Feb 2002 | A1 |
| 20020026595 | Saitou et al. | Feb 2002 | A1 |
| 20020091670 | Hitz et al. | Jul 2002 | A1 |
| 20020095557 | Constable et al. | Jul 2002 | A1 |
| 20020144057 | Li et al. | Oct 2002 | A1 |
| 20020163760 | Lindsey et al. | Nov 2002 | A1 |
| 20020166079 | Ulrich et al. | Nov 2002 | A1 |
| 20020199129 | Bohrer et al. | Dec 2002 | A1 |
| 20030004980 | Kishi et al. | Jan 2003 | A1 |
| 20030005313 | Gammel et al. | Jan 2003 | A1 |
| 20030025800 | Hunter et al. | Feb 2003 | A1 |
| 20030037211 | Winokur | Feb 2003 | A1 |
| 20030046260 | Satyanarayanan et al. | Mar 2003 | A1 |
| 20030120476 | Yates et al. | Jun 2003 | A1 |
| 20030120676 | Holavanahalli et al. | Jun 2003 | A1 |
| 20030126136 | Omoigui | Jul 2003 | A1 |
| 20030126388 | Yamagami | Jul 2003 | A1 |
| 20030135672 | Yip et al. | Jul 2003 | A1 |
| 20030149700 | Bolt | Aug 2003 | A1 |
| 20030158766 | Mital et al. | Aug 2003 | A1 |
| 20030182301 | Patterson et al. | Sep 2003 | A1 |
| 20030182350 | Dewey | Sep 2003 | A1 |
| 20030188208 | Fung | Oct 2003 | A1 |
| 20030217077 | Schwartz et al. | Nov 2003 | A1 |
| 20030225800 | Kavuri | Dec 2003 | A1 |
| 20040015731 | Chu et al. | Jan 2004 | A1 |
| 20040098244 | Dailey et al. | May 2004 | A1 |
| 20040103147 | Flesher et al. | May 2004 | A1 |
| 20040158766 | Liccione et al. | Aug 2004 | A1 |
| 20040167903 | Margolus et al. | Aug 2004 | A1 |
| 20040168034 | Homma et al. | Aug 2004 | A1 |
| 20040168057 | Margolus et al. | Aug 2004 | A1 |
| 20040181388 | Yip et al. | Sep 2004 | A1 |
| 20040181707 | Fujibayashi | Sep 2004 | A1 |
| 20040186858 | McGovern et al. | Sep 2004 | A1 |
| 20050010529 | Zalewski et al. | Jan 2005 | A1 |
| 20050044162 | Liang et al. | Feb 2005 | A1 |
| 20050063374 | Rowan et al. | Mar 2005 | A1 |
| 20050065962 | Rowan et al. | Mar 2005 | A1 |
| 20050066118 | Perry et al. | Mar 2005 | A1 |
| 20050066222 | Rowan et al. | Mar 2005 | A1 |
| 20050066225 | Rowan et al. | Mar 2005 | A1 |
| 20050076070 | Mikami | Apr 2005 | A1 |
| 20050076261 | Rowan et al. | Apr 2005 | A1 |
| 20050076262 | Rowan et al. | Apr 2005 | A1 |
| 20050076264 | Rowan et al. | Apr 2005 | A1 |
| 20050097260 | McGovern et al. | May 2005 | A1 |
| 20050108302 | Rand et al. | May 2005 | A1 |
| 20050144407 | Colgrove et al. | Jun 2005 | A1 |
| 20050182910 | Stager et al. | Aug 2005 | A1 |
| 20050240813 | Okada et al. | Oct 2005 | A1 |
| 20060010177 | Kodama | Jan 2006 | A1 |
| 20060047895 | Rowan et al. | Mar 2006 | A1 |
| 20060047902 | Passerini | Mar 2006 | A1 |
| 20060047903 | Passerini | Mar 2006 | A1 |
| 20060047905 | Matze et al. | Mar 2006 | A1 |
| 20060047925 | Perry | Mar 2006 | A1 |
| 20060047989 | Delgado et al. | Mar 2006 | A1 |
| 20060047998 | Darcy | Mar 2006 | A1 |
| 20060047999 | Passerini et al. | Mar 2006 | A1 |
| 20060143376 | Matze et al. | Jun 2006 | A1 |
| 20060259160 | Hood et al. | Nov 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 2 256 934 | Jun 2000 | CA |
| 0 845 733 | Jun 1998 | EP |
| 0 869 460 | Oct 1998 | EP |
| 1 058 254 | Dec 2000 | EP |
| 1 122 910 | Aug 2001 | EP |
| 1 233 414 | Aug 2002 | EP |
| 1333379 | Apr 2006 | EP |
| 1 671231 | Jun 2006 | EP |
| WO9903098 | Jan 1999 | WO |
| WO9906912 | Feb 1999 | WO |
| WO-0118633 | Mar 2001 | WO |
| WO-03067438 | Aug 2003 | WO |
| WO-2004084010 | Sep 2004 | WO |
| WO2005031576 | Apr 2005 | WO |
| WO2006023990 | Mar 2006 | WO |
| WO2006023991 | Mar 2006 | WO |
| WO2006023992 | Mar 2006 | WO |
| WO2006023993 | Mar 2006 | WO |
| WO2006023994 | Mar 2006 | WO |
| WO2006023995 | Mar 2006 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20080147756 A1 | Jun 2008 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60541626 | Feb 2004 | US | |
| 60542011 | Feb 2004 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10772642 | Feb 2004 | US |
| Child | 12040276 | US |
