1. Field of the Invention
The present invention relates to a storage system and a storage control device.
2. Description of the Related Art
For example, data is controlled using relatively large-scale storage systems in order to handle large quantities of various types of data in government organizations, public offices, autonomous regional bodies, business enterprises, educational organizations and the like. For instance, such storage systems are constructed from disk array devices or the like. Disk array devices are constructed by disposing numerous storage devices in the form of an array; for example, a storage region based on an RAID (redundant array of independent disks) is provided. One or more logical volumes (logical units) are formed in a physical storage region provided by a storage device group, and these logical volumes are provided to a host computer (more specifically, to a data base program operating in a host computer). The host computer (hereafter abbreviated to “host”) can perform the reading and writing of data with respect to the logical volumes by transmitting specified commands.
With the development of an informationized society and the like, there has been a continual increase in the amount of data that must be managed. Consequently, there is a demand for storage control devices that offer higher performance and a larger capacity, and new types of storage control devices have been developed one after another in order to meet this market demand. There are two conceivable methods for introducing new types of storage control devices as storage systems. One is a method in which an old type of storage control device and a new type of storage control device are completely interchanged, so that a storage system is constructed from a completely new type of storage control device (Japanese Patent Publication No. 10-508967). The other method is a method in which a new type of storage control device is added to a storage system consisting of an old type of storage device, so that new and old types of storage devices are caused to coexist.
Furthermore, a technique in which the storage region of a physical device is controlled in sector units, and a logical device is dynamically constructed in sector units, is also known (Japanese Patent Application Laid-Open No. 2001-337850).
Moreover, a technique is also known which is devised so that when a logical device is constructed from a plurality of storage devices with different capacities, an area is formed in accordance with the storage device that has the smallest capacity, and an area is formed in accordance with the smallest capacity in the case of the remaining capacity as well (Japanese Patent Application Laid-Open No. 9-288547).
In cases where a complete transition is made from an old type of storage control device to a new type of storage control device, the function and performance of the new type of storage control device can be utilized; however, the old type of storage control device cannot be effectively utilized, and the introduction costs are also increased. On the other hand, in cases where an old type of storage control device and a new type of storage control device are used together, the number of storage control devices that construct the storage system is increased, and considerable effort is required in order to control and operate both the new and old storage control devices.
Furthermore, in cases where the response of the storage device in which the old type of storage control device is installed is slow, the performance of the overall system drops as a result of this old type of storage control device being connected to the storage system. For example, such cases include cases in which the old type of storage device is a device that involves mechanical operations (such as head seeking or the like), so that the mechanical operating time is long, cases in which the capacity of the data transfer buffer of the old type of storage device is small, and the like.
Furthermore, there may also be cases in which an old type of storage device cannot be utilized “as is”, as in combinations of open type storage devices and main frames, or servers to which only storage devices with specified functions can be connected.
The present invention was devised in light of the above problems. One object of the present invention is to provide a storage system and storage control device which are devised so that different types of storage control devices such as new and old storage control devices can be caused to cooperate, thus allowing effective utilization of memory resources. Another object of the present invention is provide a storage system and storage control device which allow the utilization of an old type of storage control device as a new type of storage control device. Another object of the present invention is to provide a storage system and storage control device which are devised so that new functions can be added while utilizing the advantages of an old type of storage device. Another object of the present invention is to provide a storage system and storage control device which are devised so that the memory resources of a second storage control device can be incorporated into a first storage control device as a first virtual volume, and the storage contents of the first real volume of the first storage control device and this first virtual volume can be synchronized. Other objects of the present invention will become clear from the following description of embodiments.
In order to solve the abovementioned problems, the storage system of the present invention is a storage system which is constructed by communicably connecting a first storage control device and a second storage control device, and which performs data processing in accordance with requests from a higher device, wherein the abovementioned first storage control device comprises a first real volume, a first virtual volume that can form a copying pair with the first real volume, a first control part that respectively controls data communications between the first real volume and first virtual volume, and the higher device and second storage control device, and a synchronizing part that synchronizes the storage contents of the first real volume and the storage contents of the first virtual volume, and the second storage control device comprises a second real volume that is associated with the first virtual volume, and a second control part that respectively controls data communications between the second real volume, and the higher device and first storage control device.
For example, storage devices such as disk array devices or the like, or highly functionalized switches (fiber channel switches or the like) can be used as the storage control devices. The first storage control device respectively comprises a first real volume and a first virtual volume. The first real volume is constructed on the basis of first storage device which has a first storage control device, and the first virtual volume is constructed on the basis of a second storage device which has a second storage control device.
Specifically, the first storage control part incorporates the memory resources of the second storage control device as though these memory resources were its own memory resources, and provides these memory resources to the higher device. Furthermore, the synchronizing part synchronizes the storage contents of the first real volume and first virtual volume. Accordingly, a backup of the first real volume can be formed in the first virtual volume, and conversely, a backup of the first virtual volume can be formed in the first real volume. Here, the synchronization modes can be divided into two main types: namely, a full copying mode in which all of the storage contents are copied, and a differential copying mode in which only the differential data is copied.
In an embodiment of the present invention, the first storage control device has a first storage device, and the second storage control device has a second storage device; furthermore, the first real volume is connected to the first storage device via an intermediate storage device, and the first virtual volume is connected to the second storage device via a virtually constructed virtual intermediate storage device. Here, the intermediate storage device is a storage hierarchy which logically connects the first storage device that provides a physical storage region, and the first virtual volume. Similarly, the virtual intermediate storage device is a storage hierarchy which logically connects the second storage device that provides a physical storage region, and the first virtual volume. Furthermore, while the intermediate storage device is set in the storage region of the of the first storage device of the first storage control device, the virtual intermediate storage device is associated with the storage region of the second storage device of the second storage control device. Specifically, by mapping the second storage device in the virtual intermediate storage device, it is possible to vary the storage capacity, or to employ a stripe structure or the like.
The synchronizing part can copy the entire storage contents stored in the first real volume into the first virtual volume. Conversely, the synchronizing part can also copy the entire storage contents stored in the first virtual volume into the first real volume.
Alternatively, the synchronizing part can also copy the differential data between the storage contents of the first real volume and the storage contents of the first virtual volume into the first virtual volume. For example, after the first real volume and first virtual volume are synchronized by full copying, the copying pair consisting of the two volumes is temporarily released (split). Then, in cases where a change occurs in the storage contents of the first virtual volume as a result of a write request from the higher device, the storage contents of the two volumes can again be caused to coincide by separately controlling the this changed differential data, and copying only this differential data into the first real volume.
Here, in cases where write requests to the first real volume from the higher device are stopped, the synchronizing part can copy the differential data into the first virtual volume. As a result, the storage contents of both volumes can be matched.
In an embodiment of the present invention, the system further comprises a managing device which is communicably connected to the first storage control device and second storage control device, respectively. Furthermore, in cases where the access attribute of “write prohibited” is set in the first real volume by the managing device, the synchronizing part copies the differential data into the first virtual volume, and when the copying of the differential data is completed, the managing device can set the access attribute of the first real volume as “read and write possible”.
The function of the managing device can be constructed from a computer program. Accordingly, for example, the managing device can be constructed as a computer device that is separate from the higher device, or can be installed inside the higher device. The term “access attribute” refers to information that is used to control whether or not a given volume can be accessed. Examples of access attributes include “write prohibited (read only)” which prohibits the updating of data, “read/write possible” which allows both the reading and writing of data, “hidden” which does not respond to inquiry responses, “empty capacity 0” which responds that the state is full in the case of inquiries for empty capacity, and the like.
By starting differential copying after setting the access attribute of the volume as “write prohibited”, it is possible to prohibit updating requests (write requests) from the higher device, and to match the storage contents of the copying source volume (the first real volume in this case) and the copying destination volume (the first virtual volume in this case). Furthermore, since it is sufficient to alter only the access attribute inside the storage control device without any need to alter the setting of the higher device, data matching can be ensured by means of comparatively simple construction.
The synchronizing part can also copy differential data between the storage contents of the first real volume and the storage contents of the first virtual volume into the first real volume. Furthermore, in this case, the synchronizing part can acquire differential control information relating to the differential data from the second storage control device, and can read out differential data from the second storage control device and copy this data into the first real volume on the basis of this differential control information.
Furthermore, in cases where write requests to the second real volume from the higher device are prohibited, the synchronizing part can maintain the matching of data by copying the differential data into the first real volume.
Moreover, in cases where a managing device that is communicably connected to the first storage control device and second storage control device, respectively is provided, and the access attribute of “write prohibited” is set in the second real volume by the managing device, the synchronizing part can copy the differential data into the first real volume, and when the copying of this differential data has been completed, the managing device can also set the access attribute of the second real volume as “read and write possible”.
In an embodiment of the present invention, the storage system is a storage system in which a first storage control device and a second storage control device are communicably connected, this storage system comprising a higher device that can respectively issue access requests to the first storage control device and second storage control device, and a managing device that can communicate with the first storage control device and second storage control device, wherein the first storage control device comprises a first storage device that stores data, an intermediate storage device that is disposed in the storage region of this first storage device, a first real volume that is disposed in the storage region of this intermediate storage device, a virtual intermediate storage device that is disposed in the storage region of the second storage device of the second storage control device, a first virtual volume that is disposed in the storage region of this virtual intermediate storage device, a higher communications control part that respectively controls data communications between the higher device, and the second storage control device and managing device, a lower communications control part that controls data communications with the first storage device, a memory part that is shared by the higher communications control part and lower communications control part, and a mapping table that is stored in the memory part and that is used to map the second storage device in the virtual intermediate storage device. Furthermore, in cases where the first full copying mode that copies all of the storage contents stored in the first virtual volume into the first real volume is designated by the managing device, the higher communications control part refers to the mapping table and reads out all of the data from the second real volume, and the lower communications control part stores all of this read-out data in the first storage device. On the other hand, in cases where the second full copying mode that copies all of the storage contents stored in the first real volume into the first virtual volume is designated by the managing device, the lower communications control part reads out all of the data of the first real volume from the first storage device, and the higher communications control part refers to the mapping table and writes this read-out data into the second real volume.
Furthermore, the first storage control device and second storage control device can respectively hold differential control information that controls the differential data between the storage contents of the first real volume and the storage contents of the first virtual volume. Moreover, in cases where the first differential copying mode that copies the differential data into the first virtual volume is designated by the managing device, the lower communications control part reads out the differential data from the first storage device, and the higher communications control part refers to the mapping table and writes this read-out differential data into the second real volume. On the other hand, in cases where the second differential copying mode that copies the differential data into the first real volume is designated by the managing device, the higher communications control part reads out the differential control information controlled by the second storage control device, refers to this read-out differential control information and the mapping table, and reads out the differential data from the second real volume, and the lower communications control part stores this read-out differential data in the first storage device.
The present invention may also be understood as the invention of a storage control device. Moreover, the present invention may also be understood as a copying control method for a storage control device. Specifically, for example, this copying control method can be constructed so as to comprise the steps of mapping the second real volume of a second storage control device into the first virtual volume of a first storage control device, setting the abovementioned first virtual volume and the first real volume of the abovementioned first storage control device as a copying pair, and causing the storage contents of the abovementioned first virtual volume and the abovementioned first real volume to coincide.
There may be cases in which all or part of the means, functions and steps of present invention can be constructed as computer programs that are executed by a computer system. In case where all or part of the construction of the present invention is constructed from computer programs, these computer programs can be fixed (for example) on various types of storage media and distributed (or the like); alternatively, these computer programs can also be transmitted via communications networks.
For example, the storage system of the present embodiment can comprise a first storage device 1 which is one example of a first storage control device, a second storage device 2 which is one example of a second storage control device, a host 3 which acts as a higher device, and a managing device 4.
For example, the first storage device 1 is constructed as a disk array device. The first storage device 1 comprises three communications ports 1A through 1C; the host 3, managing device 4 and second storage device 2 are communicably connected by means of these respective communications ports. Here, for example, data communications can be performed between the respective storage devices 1 and 2, and the respective storage devices 1 and 2 and the host 3, on the basis of a fiber channel protocol.
Furthermore, for example, data communications can be performed between the respective storage devices 1 and 2 and the managing device 4 on the basis of a TCP/IP (transmission control protocol/internet protocol). However, the above are examples; the present invention is not restricted in terms of the type of protocol used.
The first storage device 1 can comprise a control part 5, an internal volume 6 used as a first real volume, and a virtual internal volume 7 used as a first virtual volume. The control part 5 respectively controls the exchange of data inside the first storage device and the exchange of data with the outside. The internal volume 6 is disposed on the basis of a physical storage device (e.g., a disk drive) disposed inside the first storage device 1. The virtual internal volume 7 has a virtual existence; the entity that stores data is present inside the second storage device 2. Specifically, the virtual internal volume 7 is constructed by mapping an external volume 9 of the second storage device 2 into a specified level of the storage hierarchy of the first storage device 1.
The control part 5 comprises a differential bit map 5A and a mapping table 5B. The differential bit map 5A comprises information that is used to control the differential between the storage contents of the internal volume 6 and the storage contents of the virtual internal volume 7 (external volume 9). When the host 3 updates the storage contents of the internal volume 6 after the internal volume 6 and virtual internal volume 7 have been synchronized, differential data 6A is generated by this updating. The differential bit map 5A comprises information that is used to control this differential data 6A. The mapping table 5B comprises information that is used to associate the external volume 9 with the virtual internal volume 7; for example, this information includes path information or the like that is used to access the external volume 9.
The second storage device 2 is communicably connected with the host 3, managing device 4 and first storage device 1, respectively via respective communications ports 2A through 2C. For example, the second storage device 2 can be constructed so that this device comprises a control part 8 and an external volume 9. The control part 8 respectively controls the exchange of data within the second storage device 2 and the exchange of data with the outside. The external volume 9 is disposed on the basis of a physical storage device disposed inside the second storage device 2. Since the volumes of the second storage device 2 are present on the outside as seen from the first storage device 1, these volumes are called external volumes here. Furthermore, the control part 8 comprises a differential bit map 8A that is used to control the differential data 9A that is generated in the external volume 9.
In the present embodiment, the internal volume 6 and virtual internal volume 7 form a copying pair. Either of these volumes may be the copying source, and either of the volumes may be the copying destination. In regard to the method used to synchronize the storage contents, there is full copying in which all of the storage contents of the copying source volume are copied into the copying destination volume, and differential copying in which the only the differential data between the copying source volume and copying destination volume is copied; either of these methods may be employed.
In cases where data is copied from the internal volume 6 into the virtual internal volume 7, the control part 5 refers to the mapping table 5B, acquires path information relating to the path to the external volume 9 which is the entity of the of the virtual internal volume 7, and transfers data to the external volume 9. Similarly, furthermore, in cases where data is copied from the virtual internal volume 7 into the internal volume 6, the control part 5 refers to the mapping table 5B, acquires path information relating to the path to the external volume 9, and writes data read out from the external volume 9 into the internal volume 6.
In the present embodiment, even in cases where the first storage device 1 incorporates the external volume 9 of the second storage device 2 as its own virtual internal volume 7, the data of the internal volume 6 and the data of the virtual internal volume 7 can be synchronized. The present embodiment will be described in greater detail below.
The host 10A comprises an HBA (host bus adapter) 11A that is used to access a first storage device 100 via a communications network CN1, and (for example) an application program 12A such as data base software or the like. Similarly, the host 10B also comprises an HBA 11B that is used to access a second storage device 200, and an application program 12B. Below, in cases where no particular distinction is made between the respective hosts 10A and 10B, these parts will be referred to simply as hosts 10, HBA 11 and application programs 12.
For example, depending on the case, an LAN (local area network), an SAN (storage area network), the internet, a dedicated circuit, a public circuit or the like can be appropriately used as the communications network CN1. For example, data communications via an LAN are performed according to a TCP/IP protocol. In cases where the hosts 10 are connected to the first storage device 100 [and second storage device] 200 via an LAN, the hosts 10 request data input and output in file units by designating file names.
On the other hand, in cases where the hosts 10 are connected to the first storage device 100 [and second storage device] 200 via an SAN, the hosts 10 request data input and output with blocks (which are the units of data control of the storage regions provided by a plurality of disk storage devices (disk drives)) in accordance with a fiber channel protocol. In cases where the communications network CN1 is an LAN, the HBA 11 is (for example) a network card corresponding to this LAN. In cases where the communications network CN1 is an SAN, the HBA 11 is (for example) a host bus adapter.
The managing device 20 is a computer device which is used to control the construction of the storage system and the like. For example, this device is operated by a user such as a system manager or the like. The managing device 20 is respectively connected to the respective storage devices 100 and 200 via a communications network CN2. As will be described later, the managing device 20 issues instructions relating to the formation of copying pairs, access attributes and the like to the respective storage devices 100 and 200.
For example, the first storage device 100 is constructed as a disk array subsystem. However, the present invention is not limited to this; the first storage device 100 can also be constructed as a highly functionalized intelligent type fiber channel switch. As will be described later, the first storage device 100 can provide the memory resources of the second storage device 200 to the host 10 as its own logical volume (logical unit).
The first storage device 100 can be divided into two main parts, i.e., a controller and a storage part 160. For example, the controller comprises a plurality of channel adapters (hereafter referred to as “CHAs”) 110, a plurality of disk adapters (hereafter referred to as “DKAs”) 120, a cache memory 130, a shared memory 140, and a connection control part 150.
Each CHA 110 performs data communications with a host 10. Each CHA 110 comprises a communications port 111 for performing communications with this host 10. The respective CHAs 110 are constructed as microcomputer systems comprising a CPU, memory and the like; these CHAs 110 interpret and execute various types of commands received from the hosts 10. Network addresses used to discriminate the respective CHAs 110 (e.g., IP addresses or WWN) are assigned to these CHAs 110, and each CHA 110 can behave separately as an NAS (network attached storage). In cases where a plurality of hosts 10 are present, the respective CHAs 110 separately receive and process requests from the respective hosts 10.
Each DKA 120 exchanges data with a disk drive 161 of the control part 160. Like the CHAs 110, each DKA 120 is constructed as a microcomputer system comprising a CPU, memory and the like. For example, the respective DKAs 120 write data received from the host 10 or read out from the second storage device 200 by the CHAs 110 into a specified address of a specified disk drive 161. Furthermore, each DKA 120 reads out data from a specified address of a specified disk drive 161, and transmits this data to a host 10 or the second storage device 200. In cases where the input or output of data is performed with the disk drive 161, each DKA 120 converts the logical address into a physical address. In cases where the disk drive 161 is controlled in accordance with an RAID, each DKA 120 accesses data according to the RAID construction. For example, each DKA 120 writes the same data into different disk drive groups (RAID groups), or performs parity calculations and writes the data and parity into the disk drive groups.
The cache memory 130 stores data received from the host 10 or second storage device 200, or stores data read out from the disk drive 161. As will be described later, a virtual intermediate storage device is constructed utilizing the storage space of the cache memory 130.
Various types of control information used in the operation of the first storage device 100 are stored in the shared memory (also called a control memory in some cases) 140. Furthermore, in addition to the setting of a work region, various types of tables such as the mapping table and the like described later are also stored in the shared memory 140.
Moreover, one or a plurality of disk drives 161 can also be used as cache disks. Furthermore, the cache memory 130 and shared memory 140 can be constructed as respectively separate memories, or some of the storage regions of the same memory can be used as cache regions, and other storage regions can be used as control regions.
The connection control part 150 connects the respective CHAs 110, the respective DKAs 120, the cache memory 130 and the shared memory 140 to each other. For example, the connection control part 150 can be constructed as a high-sped bus such as an ultra-high-speed cross-bar switch that performs data transfer by means of a high-speed switching operation.
The storage part 160 comprises a plurality of disk drives 161. For example, various types of storage disks such as hard disk drives, flexible disk drives, magnetic disk drives, semiconductor memory drives, optical disk drives or the like, or the equivalents of such drives, can be used as the disk drives 161. Furthermore, for example, different types of disks may be mixed inside the storage part 160, as in FC (fiber channel) disks, SATA (serial AT attachment) disks or the like.
Furthermore, as will be described later, a virtual internal volume 191 based on a disk drive 220 of the second storage device 200 can be formed in the first storage device 100. This virtual internal volume 191 can be provided to the host 10A in the same manner as the internal volume 190 based on the disk drive 161.
For example, the second storage device 200 comprises a controller 210 and a plurality of disk drives 220. The second storage device 200 is communicably connected with the host 10B, managing device 20 and first storage device 100, respectively via the communications port 211.
The second storage device 200 and host 10B are connected via the communications network CN1. The second storage device 200 and managing device 20 are connected via the communications network CN2. The second storage device 200 and first storage device 100 are connected via the communications network CN3. For example, the communications networks CN2 and CN3 can be constructed from SAN, LAN or the like.
The second storage device 200 may have substantially the same construction as the first storage device, or may have a simpler construction than the first storage device 100. The disk drives 220 of the second storage device 200 may be handled as internal storage devices of the first storage device 100.
Reference is now made to
As internal functions, this controller 101 comprises (for example) a first full copying control part 102, a second full copying control part 103, a first differential copying control part 104, and a second differential copying control part 105. Furthermore, various types of tables such as a mapping table T1, differential bit map T4 and the like are stored inside the shared memory 140 of the controller 101.
The first full copying control part 102 is a function that copies all of the storage contents of the virtual internal volume 191 into the internal volume 190. Conversely, the second full copying control part 103 is a function that copies all of the storage contents of the internal volume 190 into the virtual internal volume 191. Furthermore, the first differential copying control part 104 is a control that copies the differential data 192 of the internal volume 190 into the virtual internal volume 191. The second differential copying control part 105 is a function that copies the differential data 261 of the virtual internal volume 191 into the internal volume 190.
The internal volume 190 and virtual internal volume 191 are respectively disposed in the first storage device 100. The internal volume 190 is a volume that is set on the basis of the storage regions of the respective disk drives 161 that are directly governed by the first storage device 100. The virtual internal volume 191 is a volume that is set on the basis of the storage regions of the respective disk drives 220 of the second storage device 200.
The controller 210 of the second storage device 200 stores the differential bit map T4 (2) in a memory (not shown in the figures). This differential bit map T4 (2) is used to control the differential data 261 that is generated for the external volume 260 of the second storage device 200. Here, as was described above, the external volume 260 is based on the storage region of the disk drive 220, and is an internal volume with respect to the second storage device 200. However, since this volume 260 is mapped into the virtual internal volume 191 and incorporated into the first storage device 100, this volume is called the external volume 260 in the present embodiment.
The managing device 20 comprises an access attribute setting part 21. This access attribute setting part 21 is used to set access attributes for the internal volume 190 or external volume 260. The setting of access attributes can be performed manually by the user, or can be performed automatically on the basis of some type of trigger signal. The types of access attributes will be further described later.
Reference is now made to
For example, the storage structure of the first storage device 100 can be roughly divided into a physical storage hierarchy and a logical storage hierarchy. The physical storage hierarchy is constructed by a PDEV (physical device) 161 which is a physical disk. The PDEV corresponds to a disk drive.
The logical storage hierarchy can be constructed from a plurality (e.g., two types) of hierarchies. One logical hierarchy can be constructed from VDEVs (virtual devices) 162 and virtual VDEVs (hereafter called “V-VOLs”) 163 which can be handled as VDEVs 162. The other logical hierarchy can be constructed from LDEVs (logical devices) 164.
For example, the VDEVs 162 can be constructed by forming a specified number of PDEVs 161 into a group, e.g., four units as one set (3D+1P), eight units as one set (7D+1P) or the like. One RAID storage region is formed by the aggregate of the storage regions provided by the respective PDEVs 161 belonging to a group. This RAID storage region constitutes a VDEV 162.
In contrast to the construction of a VDEV 162 in a physical storage region, the V-VOL 163 is a virtual intermediate storage device which requires no physical storage region. The V-VOL 163 is not directly associated with a physical storage region, but is a receiver for the mapping of LUs (logical units) of the second storage device 200.
One or more LDEVs 164 can be respectively disposed in the VDEV 162 or V-VOL 163. For example, the LDEVs 164 can be constructed by splitting a VDEV 162 into specified lengths. In cases where the host 10 [involved] is an open type host, the host 10 can recognize the LDEV 164 as a single physical disk by mapping the LDEV 164 in the LU 165. The open type host can access a desired LDEV 164 by designating the LUN (logical unit number) or logical block address. Furthermore, in the case of a main frame type host, the LDEV 164 can be directly accessed.
The LU 165 is a device that can be recognized as an SCSI logical unit. The respective LUs 165 are connected to the host 10 via a target port 11A. One or more LDEVs 164 can be respectively associated with each LU 165. It is also possible to expand the LU size virtually by associating a plurality of LDEVs 164 with one LU 165.
The CMD (command device) 166 is a special LU that is used to transfer commands and status [information] between the I/O control program operating in the host 10 and the controller 101 (CHAs 110, DKAs 210) of the storage device 100. Commands from the host 10 are written into the CMD 166. The controller 101 of the storage device 100 executes processing corresponding to the commands that are written into the CMD 166, and writes the results of this execution into the CMD 166 as status [information]. The host 10 reads out and confirms the status [information] that is written into the CMD 166, and then writes the processing contents that are to be executed next into the CMD 166. Thus, the host 10 can issue various types of instructions to the storage device 100 via the CMD 166.
Furthermore, the commands received from the host 10 can also be processed without being stored in the CMD 166. Moreover, the CMD can also be formed as a virtual device without defining the actual device (LU), and can be constructed so as to receive and process commands from the host 10. Specifically, for example, the CHAs 110 write the commands received from the host 10 into the shared memory 140, and the CHAs 110 or DKAs 120 process the commands stored in this shared memory 140. The processing results are written into the shared memory 140, and are transmitted to the host 10 from the CHAs 110.
The second storage device 200 is connected to the external initiator port (external port) 111B of the first storage device 100 via the communications network CN3.
The second storage device 200 comprises a plurality of PDEVs 220, VDEVs 230 that are set in storage regions provided by the PDEVs 220, and one or more LDEVs 240 that can be set in the VDEVs 230. Each LDEV 240 is respectively associated with an LU 250.
Furthermore, in the present embodiment, the LUs 250 (i. e., the LDEVs 240) of the second storage device 200 are mapped into a V-VOL 163 which is a virtual intermediate storage device so that these LUs 250 can also be used from the first storage device 100.
For example, in
Furthermore, the VDEVs 162 and V-VOLs 163 can use an RAID construction. Specifically, one disk drive 161 can be divided into a plurality of VDEVs 162 and V-VOLs 163 (slicing), or one VDEV 162 or V-VOL 163 can be formed from a plurality of disk drives 161 (striping).
Furthermore, the “LDEV 1” or “LDEV 2” of the first storage device 100 corresponds to internal volume 190 in
Reference is now made to
For example, the mapping table T1 can be constructed by respectively establishing a correspondence between the VDEV numbers used to discriminate the VDEVs 162 and V-VOLs 163 and information relating to the external disk drives 220.
For example, the external device information can be constructed so that this information includes device discriminating information, storage capacities of the disk drives 220, information indicating the type of device (tape type devices, disk type devices or the like) and path information indicating the paths to the disk drives 220. This path information can be constructed so as to include discriminating information (WWN) specific to the respective communications ports 211, and LUN numbers used to discriminate the LUs 250.
Furthermore, the values of the device discriminating information, WWN and the like shown in
By using a mapping table T1 such as that shown in
Furthermore, as is also true of the other tables shown below, the volume numbers and the like shown in the table are examples used to illustrate the table construction; these values do not particularly correspond to the other constructions shown in
The conditions of data conversion using these various types of tables will be described with reference to
The first storage device 100 converts the data that is input for LDEV use (LUN #+LBA) into data for VDEV use on the basis of the first conversion table T2 shown in
For example, this first conversion table T2 is constructed by associating LUN numbers (LUN #), LDEV numbers (LDEV #) and maximum slot numbers that correspond to correspond to these LUNs, VDEV (including V-VOL) numbers (VDEV #) and maximum slot numbers that correspond to these LDEVs and the like. As a result of reference being made to this first conversion table T2, the data from the host 10 (LUN #+LBA) is converted into VDEV data (VDEV #+SLOT #+SUBLOCK #).
Next, the first storage device 100 refers to the second conversion table T3 shown in
In the second conversion table T3, for example, VDEV numbers (VDEV #), the numbers of initiator ports used to transmit data from the VDEVs to the second storage device 200, WWN used to specify the communications ports that are the data transfer destinations and LUNs that can be accessed via these communications ports are associated.
On the basis of this second conversion table T3, the first storage device 100 converts the address information of the data that is to be stored into the format of initiator port number #+WWN+LUN #+LBA. The data whose address information has thus been altered reaches the designated communications port 211 from the designated initiator port via the communications network CN3. Then, the data is stored in a specified place in the LDEV.
c) shows another second conversion table T3a. This conversion table T3a is used in cases where a stripe or RAID is applied to VDEVs (i.e., V-VOLs) originating in an external disk drive 220. The conversion table T3a is constructed by associating VDEV numbers (VDEV #), stripe sizes, RAID levels, numbers used to discriminate the second storage device 200 (SS # (storage system numbers)), initiator port numbers and WWN and LUN numbers of the communications ports 211.
In the example shown in
For example, the differential bit map T4 can be constructed by associating updating flag information indicating the status as to whether or not updating has been performed with each logical track of the disk drives 161 constituting the internal volume 190. One logical track corresponds to three cache segments, and has size of 48 kB or 64 kB.
For example, the saving destination address control table can be constructed by associating with each logical track unit a saving destination address which indicates where the data stored on this track is saved. Furthermore, in the tables T4 and T5, the control units are not limited to track units. For example, other control units such as slot units, LBA units or the like can also be used.
Here, the “pair form” status is a state in which initial copying (full copying) from the copying source volume to the copying destination volume has been performed, so that a copying pair is formed. The “pair split” status is a state in which the copying source volume and copying destination volume are separated after the copying pair has been forcibly synchronized. The “resynchronize” status is a state in which the storage contents of the copying source volume and copying destination volume are resynchronized and a copying pair is formed after the two volumes have been separated.
Examples of access attributes include “read/write possible”, “write prohibited (read only)”, “read/write impossible”, “empty capacity 0”, “copying destination setting impossible” and “hidden”.
Here, “read/write possible” indicates a state in which reading and writing from and into the volume in question are possible. “Write prohibited” indicates a state in which writing into the volume in question is prohibited, so that only read-out is permitted. “Read/write impossible” indicates a state in which writing and reading into and from the volume are prohibited. “Empty capacity 0” indicates a state in which a response of remaining capacity 0 (full) is given in reply to inquiries regarding the remaining capacity of the volume even in cases where there is actually some remaining capacity. “Copying destination setting impossible” indicates a state in which the volume in question cannot be set as the copying destination volume (secondary volume). “Hidden” indicates a state in which the volume in question cannot be recognized from the initiator. Furthermore, as was already mentioned above, the LUNs in the table are numbers used for purposes of description; these numbers in themselves have no particular significance.
Next, the operation of the present embodiment will be described. First,
The first storage device 100 logs into the second storage device 200 via the initiator port of the CHA 110 (S1). Logging in is completed by the second storage device 200 sending back a response to the logging in of the first storage device 100 (S2). Next, for example, the first storage device 100 transmits an inquiry command determined by the SCSI (small computer system interface) to the second storage device 200, and requests a response regarding details of the disk drives 220 belonging to the second storage device 200 (S3).
The inquiry command is used to clarify the type and construction of the inquiry destination device; this makes it possible to pass through the hierarchy of the inquiry destination device and grasp the physical structure of this inquiry destination device. By using such an inquiry command, for example, the first storage device 100 can acquire information such as the device name, device type, manufacturing serial number (product ID), LDEV number, various types of version information, vendor ID and the like from the second storage device 200 (S4). The second storage device 200 responds by transmitting the information for which an inquiry was made to the first storage device 100 (S5).
The first storage device 100 registers the information acquired from the second storage device 200 in the mapping table T1 (S6). The first storage device 100 reads out the storage capacity of the disk drive 220 from the second storage device 200 (S7). In response to an inquiry from the first storage device 100, the second storage device 200 sends back the storage capacity of the disk drive 220 (S8), and returns a response (S9). The first storage device 100 registers the storage capacity of the disk drive 220 in a specified place in the mapping table T1 (S10).
The mapping table T1 can be constructed by performing the above processing. In cases where the input and output of data are performed with the external disk drive 220 (external LUN, i.e., external volume 260) mapped into the V-VOL of the first storage device 100, address conversion and the like are performed with reference to the other conversion tables T2 and T3 described with reference to
Next, the input and output of data between the first storage device 100 and second storage device 200 will be described.
The host 10 can write data into a logical volume (LDEV) that has access authorization. For example, by using procedures such as zoning that sets a virtual SAN sublet in the SAN or LUN masking in which the host 10 holds a list of accessible LUNs, it is possible to set the host 10 so that the host 10 can access only specified LDEVs.
In cases where the LDEV into which the host 10 is to write data is connected via a VDEV to a disk drive 161 which is in internal storage device, data is written by ordinary processing. Specifically, the data from the host 10 is temporarily stored in the cache memory 130, and is then stored in a specified address of a specified disk drive 161 from the cache memory 130 via the DKA 120. In this case, the DKA 120 converts the logical address into a physical address. Furthermore, in the case of a raid construction, the same data is stored in a plurality of disk drives 161 or the like.
On the other hand, in cases where the LDEV into which the host 10 is to write data is connected to an external disk drive 220 via a V-VOL, the flow is as shown in
The host 10 indicates an LDEV number that specifies the LDEV that is the object of writing and a WWN that specifies the communications port that is used to access this LDEV, and issues a write command (write) (S21). When the first storage device 100 receives a write command from the host 10, the first storage device 100 produces a write command for transmission to the second storage device 200, and transmits this command to the second storage device 200 (S22). The first storage device 100 alters the address information and the like contained in the write command received from the host 10 so as to match the external volume 260, thus producing a new write command.
The host 10 transmits the write data to the to the first storage device 100 (S23). The write data received by the first storage device 100 is transferred to the second storage device 200 (S26) from the LDEV via the V-VOL (S24). Here, at the point in time at which the data from the host 10 is stored in the cache memory 130, the first storage device 100 sends back a response (good) indicating the completion of writing to the host 10.
At the point in time at which the write data is received from the first storage device 100 (or the point in time at which writing into the disk drive 220 is completed), the second storage device 200 transmits a writing completion report to the first storage device 100 (S26). Specifically, the time at which the completion of writing is reported to the host 10 by the first storage device 100 (S25) and the time at which the data is actually stored in the disk drive 220 are different (asynchronous system). Accordingly, the host 10 is released from data write processing before the write data is actually stored in the disk drive 220, so that the host 10 can perform other processing.
Reference will now be made to
The flow in cases where data is read out from the external volume 260 of the second storage device 200 will be described with reference to
First, the host 10 designates a communications port 111 and transmits a data read-out command to the first storage device 100 (S31). When the first storage device 100 receives a read command, the first storage device 100 produces a read command in order to read out the requested data from the second storage device 200.
The first storage device 100 transmits the produced read command to the second storage device 200 (S32). In accordance with the read command received from the first storage device 100, the second storage device 200 reads out the requested data from the disk drive 220, transmits this read-out data to the first storage device 100 (S33), and reports that read-out was normally completed (S35). As is shown in
The first storage device 100 reads out the data stored in the cache memory 130, performs address conversion, transmits the data to the host 10 via the LUN 103 or the like (S36), and issues a read-out completion report (S37). In the series of processing performed in these data read-outs, the conversion operation described with reference to
In
Next, the method used to synchronize the storage contents between the internal volume 190 and virtual internal volume 191 (whose substance is the external volume 260) will be described.
The managing device 20 instructs the first storage device 100 to execute the first full copying mode (S41). The CHA 110 that receives this instruction refers to the mapping table T1 stored in the shared memory 140 (S42), and acquires path information for the external volume 260 which is the copying destination volume. The CHA 110 issues a read command to the second storage device 200 (S43), and requests the read-out of the data that is stored in the external volume 260.
In response to the read command from the first storage device 100, the second storage device 200 reads out data from the external volume 260 (S44), and transmits this read-out data to the first storage device 100 (S45).
When the CHA 110 receives the data from the second storage device 200, the CHA 110 stores this received data in the cache memory 130 (S46). Furthermore, for example, the CHA 110 requests the execution of destage processing from the DKA 120 by writing a write command into shared memory 140 (S47).
The DKA 120 occasionally refers to the shared memory 140, and when the DKA 120 discovers an unprocessed write command, the DKA 120 reads out the data stored in the cache memory 130, performs processing such as address conversion and the like, and writes this data into a specified disk drive 161 (S48).
Thus, all of the storage contents of the external volume 260 which is the copying source volume can be copied into the internal volume 190 which is the copying destination volume, so that the storage contents of both volumes are caused to coincide.
In response to this staging request, the DKA 120 reads out the data of the internal volume 190 from the disk drive 161, and stores this data in the cache memory 130 (S54). Furthermore, the DKA 120 request that the CHA 110 issue a write command (S55).
On the basis of the path information acquired in S52, the CHA 110 issues a write command to the second storage device 200 (S56). Next, the CHA 110 transmits write data to the second storage device 200 (S57).
The second storage device 200 receives the write data from the first storage device 100 (S58), and stores this data in a specified disk drive 220 (S59). Thus, the storage contents of the internal volume 190 which is the copying source volume can be copied into the external volume 260 which is the copying destination volume, so that the storage contents of both volumes can be caused to coincide.
The host 10A executes updating I/O for the internal volume 190 (S63). The CHA 110 stores the write data received from the host 10A in the cache memory 130 (S64), and sends a response to the host 10A indicating that processing of the write command has been completed (S65).
Furthermore, the CHA 110 respectively updates the differential bit map T4 and the differential data 192 (S66), and requests that the DKA 120 execute destage processing (S67). The DKA 120 stores the write data generated by the updating I/O in the disk drive 161 (S68).
Prior to the initiation of differential copying, the updating I/O from the host 10A is stopped (S69). For example, this stopping of the I/O can be accomplished manually by the user. Furthermore, the managing device 20 alters the access attribute of the internal volume 190 from “read/write possible” to “write prohibited” (S70). Although the issuing of updating I/O by the host 10A is already stopped, further variation of the storage contents of the internal volume 190 can be prevented in advance by altering the access attribute to “write prohibited”.
Then, the managing device 20 instructs the first storage device 100 to execute first differential copying (S71). The CHA 110 that receives this instruction refers to the mapping table T1 (S72), and acquires path information for the external volume 260. Furthermore, the CHA 110 refers to the differential bit map T4 (S73), and requests that the DKA 120 perform destaging of the differential data 192 (S74).
The DKA 120 reads out the differential data 192 produced for the internal volume 190 from the disk drive 161, and stores this data in the cache memory 130 (S75). Then, the DKA 120 requests that the CHA 110 issue a write command (S76).
The CHA 110 issues a write command to the second storage device 200 (S77), and transmits write data (the differential data 192) to the second storage device 200 (S78). The second storage device 200 stores the received write data in the external volume 260. As a result, the storage contents of the external volume 260 and internal volume 190 coincide. Then, the managing device 20 alters the access attribute of the internal volume 190 from “write prohibited” to “read/write possible” (S79).
Then, when the host 10B accesses the external volume 260 and issues updating I/O (S83), the second storage device 200 writes the write data into the disk drive 220 (S84), and respectively updates the differential data 261 and differential bit map T4 (2) (S85).
When differential copying is initiated, the managing device 20 alters the access attribute of the external volume 260 from “read/write possible” to “write prohibited” (S86), thus prohibiting updating of the external volume 260; then, the managing device 20 instructs the first storage device 100 to initiate second differential copying (S87).
The CHA 110 that receives the instruction to initiate differential copying requests that the second storage device 200 to transfer the differential bit map T4 (2) (S88). Since the contacts of the differential data 261 generated in the external volume 260 are controlled by the second storage device 200, the first storage device 100 acquires the differential bit map T4 (2) from the second storage device 200 (S89).
Furthermore, in this embodiment, a construction is used in which commands and data are directly exchanged between the first storage device 100 and second storage device 200. However, the present invention is not limited to this; for example, it would also be possible to exchange data such as the differential bit map and the like between the respective storage devices 100 and 200 via the managing device 20.
The CHA 110 refers to the mapping table T1 (S90), and acquires path information indicating the path to the external volume 260. Then, the CHA 110 requests the transfer of the differential data 261 by issuing a read command to the second storage device 200 (S91).
In response to the read command from the first storage device 100, the second storage device 200 transmits the differential data 261 to the first storage device 100 (S92). Then, the CHA 110 that receives this differential data 261 stores the differential data 261 in the cache memory 130 (S93) The CHA 110 requests that the DKA 120 perform destage processing of the differential data 261 (S94). Then, the DKA 120 reads out the differential data 261 stored in the cache memory 130, and writes the data constituting the internal volume 190 into the disk drive 161 (S95). As a result, the storage contents of the external volume 260 and internal volume 190 coincide.
In the present embodiment, as was described in detail above, the external volume 260 can be handled as though this volume were a logical volume inside the first storage device 100 by mapping the external disk drive 220 into the V-VOL. Accordingly, even in cases where the second storage device 200 is an old type device that cannot be directly connected to the host 10, the memory resources of the old type device can be reutilized as memory resources of the first storage device 100, and can be provided to the host 10, by interposing a new type first storage device 100. As a result, the old type storage device 200 can be connected to a new type storage device 100, and the memory resources can be effectively utilized.
Furthermore, in cases where the first storage device 100 is a high-performance, highly functional new type device, the low performance of the second storage device can be hidden by the high-performance computer resources (cache capacity, CPU processing speed and the like) of the first storage device 100, so that high-performance services can be provided to the host 10 using a virtual internal volume that utilizes the disk drive 220. Furthermore, functions such as (for example) striping, expansion, splitting, RAID and the like can be added to an external volume 260 constructed in the disk drive 220, and can be used. Accordingly, compared to cases in which an external volume is directly mapped into an LUN, the degree of freedom of utilization is increased so that convenience of use is improved.
In the present embodiment, in addition to these effects, the storage contents can be synchronized between the internal volume 190 and virtual internal volume 191 (external volume 260). Accordingly, a backup of the internal volume 190 can be formed in the virtual internal volume 191, or conversely, a backup of the virtual internal volume 191 can be formed in the internal volume 190, so that the convenience is even further improved.
Furthermore, in the present embodiment, since both a full copying mode and a differential copying mode can be performed, efficient copying can be performed in accordance with the conditions.
Furthermore, in the present embodiment, a construction is used in which the storage contents of the copying source volume are fixed by altering the access attribute to “write prohibited”. Accordingly, volume copying can be executed without particularly altering the processing contents in the host 10.
A second embodiment of the present invention will be described with reference to
In the present embodiment, the first storage device 100 comprises a third storage device 300 in addition to a second storage device 200. Like the second storage device 200, this third storage device 300 is a device that is externally connected to the first storage device 100. Like the second storage device 200, the third storage device 300 comprises (for example) PDEVs 320, VDEVs 330, LDEVs 349, LUs 350, targets 311 and the like. In regard to the construction of the third storage device 300, the construction of the second storage device 200 can be employed; since this construction is not the gist of the present invention, details will be omitted. However, the second storage device 200 and third storage device 300 need not have the same structure.
The first storage device 100 does not comprise PDEVs 161 which are physical storage devices, and does not comprise real volumes (internal volumes). The first storage device 100 operates as a virtualization storage device and comprises only “LDEV 1” and “LDEV 2”, which are virtual internal volumes. Accordingly, the first storage device 100 need not be a disk array device; for example, this first storage device 100 may be an intelligent type switch comprises a computer system.
The first virtual internal volume “LDEV 1” 164 is connected to “LDEV 1” 240, which is a real volume of the second storage device 200, via “V-VOL” 163. The second virtual internal volume “LDEV 2” 164 is connected to “LDEV 1” 340, which is a real volume of the third storage device 300, via “V-VOL 2” 163.
Furthermore, in the present embodiment, the system is devised so that full copying and differential copying are performed between the first virtual internal volume “LDEV 1” and the second virtual internal volume “LDEV 2” inside the first storage device 100.
A third embodiment will be described with reference to
For example, in cases where copying pairs are set in the storage system, the user logs into the managing device 20, and calls up a control screen such as that shown in
A plurality of different types of control menus M1 through M3 can be set on the control screen. For example, these control menus M1 through M3 can be constructed as tab type switching menus. For instance, the menu M1 is a menu that is used to perform various types of LU operations such as production of volumes or the like. The menu M2 is a menu that is used to perform communications port operations. The menu M3 is a menu that is used to perform volume copying operations between the storage devices described in the abovementioned embodiments.
For example, the menu M3 can be constructed so that this menu includes a plurality of screen regions G1 through G5. The screen region G1 is used to select the storage device (subsystem) that performs the setting of copying pairs. The conditions of the set copying pairs are displayed in the screen region G2. For instance, the copying source volume (P-VOL), copying destination volume (S-VOL), emulation type, capacity, copying status, progression, copying speed and the like can be displayed in the screen region G2.
For instance, using a pointing device such as a mouse or the like, the user can select two copying pairs displayed in the screen region G2; furthermore, the user can display the submenu M4 by right clicking [with the mouse]. The user can designate the synchronization of volumes or dissolution of pairs by means of the submenu M4.
In the screen region G3, either internal volumes inside the first storage device 100 or external volumes inside the second storage device 200 can be exclusively selected as the volumes of the copying pair. In the figures, a case is shown in which an internal volume is selected as the copying source volume. An internal volume or external volume can be designated as either the copying source volume or copying destination volume.
Preset values can be displayed in the screen region G4. Operation states can be displayed in the screen region G5. When the setting of the copying pair has been completed, the user can cause alterations in the construction to be reflected by operating an application button B1. In cases where the content of this setting is to be canceled, the user operates a cancel button B2. The abovementioned screen construction is an example; the present invention is not limited to this construction.
Furthermore, the present invention is not limited to the respective embodiments described above. A person skilled in the art can make various additions, alterations and the like within the scope of the present invention.
Number | Date | Country | Kind |
---|---|---|---|
2004-312358 | Oct 2004 | JP | national |
This is a continuation of U.S. application Ser. No. 11/016,806, filed Dec. 21, 2004, which relates to U.S. application Ser. No. 11/697,777, filed Apr. 9, 2007. This application relates to and claims priority from Japanese Patent Application No. 2004-312358, filed on Oct. 27, 2004. The entirety of the contents and subject matter of all of the above is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
3771137 | Barner et al. | Nov 1973 | A |
4025904 | Adney et al. | May 1977 | A |
4710868 | Cocke et al. | Dec 1987 | A |
5155845 | Beal et al. | Oct 1992 | A |
5170480 | Mohan et al. | Dec 1992 | A |
5307481 | Shimazaki et al. | Apr 1994 | A |
5379418 | Shimazaki et al. | Jan 1995 | A |
5408465 | Gusella et al. | Apr 1995 | A |
5459857 | Ludlam et al. | Oct 1995 | A |
5504882 | Chai et al. | Apr 1996 | A |
5544347 | Yanai et al. | Aug 1996 | A |
5548712 | Larson et al. | Aug 1996 | A |
5555371 | Duyanovich et al. | Sep 1996 | A |
5574950 | Hathorn et al. | Nov 1996 | A |
5680580 | Beardsley et al. | Oct 1997 | A |
5680640 | Ofek et al. | Oct 1997 | A |
5720029 | Kern et al. | Feb 1998 | A |
5734818 | Kern et al. | Mar 1998 | A |
5742792 | Yanai et al. | Apr 1998 | A |
5758118 | Choy et al. | May 1998 | A |
5799323 | Mosher et al. | Aug 1998 | A |
5809285 | Hilland | Sep 1998 | A |
5835953 | Ohran | Nov 1998 | A |
5835954 | Duyanovich et al. | Nov 1998 | A |
5870537 | Kern et al. | Feb 1999 | A |
5895485 | Loechel et al. | Apr 1999 | A |
5901327 | Ofek | May 1999 | A |
5917723 | Binford | Jun 1999 | A |
5933653 | Ofek | Aug 1999 | A |
5956750 | Yamamoto et al. | Sep 1999 | A |
5974563 | Beeler | Oct 1999 | A |
5978890 | Ozawa et al. | Nov 1999 | A |
5995980 | Olson et al. | Nov 1999 | A |
6012123 | Pecone et al. | Jan 2000 | A |
6044444 | Ofek | Mar 2000 | A |
6052758 | Crockett et al. | Apr 2000 | A |
6092066 | Ofek | Jul 2000 | A |
6098129 | Fukuzawa et al. | Aug 2000 | A |
6101497 | Ofek | Aug 2000 | A |
6108748 | Ofek et al. | Aug 2000 | A |
6148383 | Micka et al. | Nov 2000 | A |
6157991 | Arnon | Dec 2000 | A |
6173374 | Heil et al. | Jan 2001 | B1 |
6173377 | Yanai et al. | Jan 2001 | B1 |
6178427 | Parker | Jan 2001 | B1 |
6195730 | West | Feb 2001 | B1 |
6209002 | Gagne et al. | Mar 2001 | B1 |
6219753 | Richardson | Apr 2001 | B1 |
6230239 | Sakaki et al. | May 2001 | B1 |
6237008 | Beal et al. | May 2001 | B1 |
6240486 | Ofek et al. | May 2001 | B1 |
6240494 | Nagasawa et al. | May 2001 | B1 |
6247099 | Skazinski et al. | Jun 2001 | B1 |
6247103 | Kern et al. | Jun 2001 | B1 |
6253295 | Beal et al. | Jun 2001 | B1 |
RE37305 | Chang et al. | Jul 2001 | E |
6282610 | Bergsten | Aug 2001 | B1 |
6308283 | Galipeau | Oct 2001 | B1 |
6324654 | Wahl et al. | Nov 2001 | B1 |
8341329 | LeCrone at al. | Jan 2002 | |
6356977 | Ofek et al. | Mar 2002 | B2 |
6360306 | Bergsten | Mar 2002 | B1 |
6363462 | Bergsten | Mar 2002 | B1 |
6393538 | Murayama | May 2002 | B2 |
6397307 | Ohran | May 2002 | B2 |
6408370 | Yamamoto et al. | Jun 2002 | B2 |
6442706 | Wahl et al. | Aug 2002 | B1 |
6446141 | Nolan et al. | Sep 2002 | B1 |
6446175 | West et al. | Sep 2002 | B1 |
6446176 | West et al. | Sep 2002 | B1 |
6453354 | Jiang et al. | Sep 2002 | B1 |
6457109 | Milillo et al. | Sep 2002 | B1 |
6457139 | D'Errico et al. | Sep 2002 | B1 |
6460055 | Midgley et al. | Oct 2002 | B1 |
6463501 | Kern et al. | Oct 2002 | B1 |
6467034 | Yanaka | Oct 2002 | B1 |
6477627 | Ofek | Nov 2002 | B1 |
6480934 | Hino et al. | Nov 2002 | B1 |
6484173 | O'Hare et al. | Nov 2002 | B1 |
6487645 | Clark et al. | Nov 2002 | B1 |
6490659 | McKean et al. | Dec 2002 | B1 |
6496908 | Kamvysselis et al. | Dec 2002 | B1 |
6523096 | Sanada et al. | Feb 2003 | B2 |
6526487 | Ohran et al. | Feb 2003 | B2 |
6529976 | Fukuzawa et al. | Mar 2003 | B1 |
6553408 | Merrell et al. | Apr 2003 | B1 |
6560673 | Elliott | May 2003 | B2 |
6587935 | Ofek | Jul 2003 | B2 |
6598134 | Ofek et al. | Jul 2003 | B2 |
6622152 | Sinn et al. | Sep 2003 | B1 |
6622220 | Yoshida et al. | Sep 2003 | B2 |
6625623 | Midgley et al. | Sep 2003 | B1 |
6640278 | Nolan et al. | Oct 2003 | B1 |
6640291 | Fujibayashi | Oct 2003 | B2 |
6643671 | Milillo et al. | Nov 2003 | B2 |
6647387 | McKean et al. | Nov 2003 | B1 |
6647414 | Eriksson et al. | Nov 2003 | B1 |
6647474 | Yanai et al. | Nov 2003 | B2 |
6647476 | Nagasawa et al. | Nov 2003 | B2 |
6654830 | Taylor et al. | Nov 2003 | B1 |
6654831 | Otterness et al. | Nov 2003 | B1 |
6662197 | LeCrone et al. | Dec 2003 | B1 |
6675258 | Bramhall et al. | Jan 2004 | B1 |
6681303 | Watabe et al. | Jan 2004 | B1 |
6681339 | McKean et al. | Jan 2004 | B2 |
6684310 | Anzai et al. | Jan 2004 | B2 |
6697367 | Halstead et al. | Feb 2004 | B1 |
6708232 | Obara | Mar 2004 | B2 |
6718404 | Reuter et al. | Apr 2004 | B2 |
6745281 | Saegusa | Jun 2004 | B1 |
6772315 | Perego | Aug 2004 | B1 |
6799255 | Blumenau et al. | Sep 2004 | B1 |
6804676 | Bains | Oct 2004 | B1 |
6816948 | Kitamura et al. | Nov 2004 | B2 |
6826778 | Bopardikar et al. | Nov 2004 | B2 |
6851020 | Matsumoto et al. | Feb 2005 | B2 |
6857057 | Nelson et al. | Feb 2005 | B2 |
6876656 | Brewer et al. | Apr 2005 | B2 |
6883064 | Yoshida et al. | Apr 2005 | B2 |
6883122 | Maple et al. | Apr 2005 | B2 |
6922761 | O'Connell et al. | Jul 2005 | B2 |
6941322 | Bills et al. | Sep 2005 | B2 |
6959369 | Ashton et al. | Oct 2005 | B1 |
6968349 | Owen et al. | Nov 2005 | B2 |
6973549 | Testardi | Dec 2005 | B1 |
6976103 | Watanabe et al. | Dec 2005 | B1 |
6976134 | Lolayekar et al. | Dec 2005 | B1 |
7143262 | Serizawa | Nov 2006 | B2 |
7373472 | Bhasin et al. | May 2008 | B2 |
20010029570 | Yamamoto et al. | Oct 2001 | A1 |
20010050915 | O'Hare et al. | Dec 2001 | A1 |
20010052018 | Yokokura | Dec 2001 | A1 |
20010054133 | Murotani et al. | Dec 2001 | A1 |
20020003022 | Csida et al. | Jan 2002 | A1 |
20020004857 | Arakawa et al. | Jan 2002 | A1 |
20020004890 | Ofek et al. | Jan 2002 | A1 |
20020019908 | Reuter et al. | Feb 2002 | A1 |
20020019920 | Reuter et al. | Feb 2002 | A1 |
20020019922 | Reuter et al. | Feb 2002 | A1 |
20020019923 | Reuter et al. | Feb 2002 | A1 |
20020026558 | Reuter et al. | Feb 2002 | A1 |
20020029326 | Reuter et al. | Mar 2002 | A1 |
20020065864 | Hartsell et al. | May 2002 | A1 |
20020087544 | Selkirk et al. | Jul 2002 | A1 |
20020087751 | Chong | Jul 2002 | A1 |
20020103889 | Markson et al. | Aug 2002 | A1 |
20020112113 | Karpoff et al. | Aug 2002 | A1 |
20020124108 | Terrell et al. | Sep 2002 | A1 |
20020133511 | Hostetter et al. | Sep 2002 | A1 |
20020133735 | McKean et al. | Sep 2002 | A1 |
20020143888 | Lisiecki et al. | Oct 2002 | A1 |
20020156887 | Hashimoto | Oct 2002 | A1 |
20020156984 | Padovano | Oct 2002 | A1 |
20020156987 | Gajjar et al. | Oct 2002 | A1 |
20020178328 | Honda et al. | Nov 2002 | A1 |
20020178335 | Selkirk et al. | Nov 2002 | A1 |
20020184463 | Arakawa et al. | Dec 2002 | A1 |
20020188592 | Leonhardt et al. | Dec 2002 | A1 |
20020194428 | Green | Dec 2002 | A1 |
20020194523 | Ulrich et al. | Dec 2002 | A1 |
20030014432 | Teloh et al. | Jan 2003 | A1 |
20030014433 | Teloh et al. | Jan 2003 | A1 |
20030037071 | Harris et al. | Feb 2003 | A1 |
20030051109 | Cochran | Mar 2003 | A1 |
20030051111 | Nakano et al. | Mar 2003 | A1 |
20030056038 | Cochran | Mar 2003 | A1 |
20030074378 | Midgley et al. | Apr 2003 | A1 |
20030074600 | Tamatsu et al. | Apr 2003 | A1 |
20030079018 | Lolayekar et al. | Apr 2003 | A1 |
20030079019 | Lolayekar et al. | Apr 2003 | A1 |
20030084075 | Balogh et al. | May 2003 | A1 |
20030093541 | Lolayekar et al. | May 2003 | A1 |
20030093567 | Lolayekar et al. | May 2003 | A1 |
20030097607 | Bessire | May 2003 | A1 |
20030101228 | Busser et al. | May 2003 | A1 |
20030105931 | Weber et al. | Jun 2003 | A1 |
20030115218 | Bobbitt et al. | Jun 2003 | A1 |
20030115432 | Biessener et al. | Jun 2003 | A1 |
20030126327 | Pesola et al. | Jul 2003 | A1 |
20030126388 | Yamagami | Jul 2003 | A1 |
20030145168 | LeCrone et al. | Jul 2003 | A1 |
20030145169 | Nagasawa et al. | Jul 2003 | A1 |
20030158999 | Hauck et al. | Aug 2003 | A1 |
20030163553 | Kitamura et al. | Aug 2003 | A1 |
20030167419 | Yanai et al. | Sep 2003 | A1 |
20030172069 | Uchiyama et al. | Sep 2003 | A1 |
20030182525 | O'Connell et al. | Sep 2003 | A1 |
20030189936 | Terrell et al. | Oct 2003 | A1 |
20030200387 | Urabe et al. | Oct 2003 | A1 |
20030204479 | Bills et al. | Oct 2003 | A1 |
20030204597 | Arakawa et al. | Oct 2003 | A1 |
20030204700 | Biessener et al. | Oct 2003 | A1 |
20030212854 | Kitamura et al. | Nov 2003 | A1 |
20030212860 | Jiang et al. | Nov 2003 | A1 |
20030217031 | Owen et al. | Nov 2003 | A1 |
20030220935 | Vivian et al. | Nov 2003 | A1 |
20030221077 | Ohno et al. | Nov 2003 | A1 |
20030229645 | Mogi et al. | Dec 2003 | A1 |
20030229764 | Ohno et al. | Dec 2003 | A1 |
20040003022 | Garrison et al. | Jan 2004 | A1 |
20040028043 | Maveli et al. | Feb 2004 | A1 |
20040030703 | Bourbonnais et al. | Feb 2004 | A1 |
20040049553 | Iwamura et al. | Mar 2004 | A1 |
20040054850 | Fisk | Mar 2004 | A1 |
20040054866 | Blumenau et al. | Mar 2004 | A1 |
20040059738 | Tarbell | Mar 2004 | A1 |
20040064610 | Fukuzawa et al. | Apr 2004 | A1 |
20040064641 | Kodama | Apr 2004 | A1 |
20040068637 | Nelson et al. | Apr 2004 | A1 |
20040073831 | Yanai et al. | Apr 2004 | A1 |
20040098547 | Ofek et al. | May 2004 | A1 |
20040103261 | Honda et al. | May 2004 | A1 |
20040111485 | Mimatsu et al. | Jun 2004 | A1 |
20040123180 | Soejima et al. | Jun 2004 | A1 |
20040139237 | Rangan et al. | Jul 2004 | A1 |
20040143832 | Yamamoto et al. | Jul 2004 | A1 |
20040148443 | Achiwa et al. | Jul 2004 | A1 |
20040158652 | Obara | Aug 2004 | A1 |
20040158673 | Matsunami et al. | Aug 2004 | A1 |
20040172510 | Nagashima et al. | Sep 2004 | A1 |
20040230980 | Koyama et al. | Nov 2004 | A1 |
20040250021 | Honda et al. | Dec 2004 | A1 |
20040260875 | Murotani et al. | Dec 2004 | A1 |
20040260966 | Kaiya et al. | Dec 2004 | A1 |
20050010734 | Soejima et al. | Jan 2005 | A1 |
20050010743 | Tremblay et al. | Jan 2005 | A1 |
20050033878 | Pangal et al. | Feb 2005 | A1 |
20050038968 | Iwamura et al. | Feb 2005 | A1 |
20050050115 | Kekre | Mar 2005 | A1 |
20050055501 | Guha et al. | Mar 2005 | A1 |
20050081009 | Williams et al. | Apr 2005 | A1 |
20050138184 | Amir et al. | Jun 2005 | A1 |
20060090048 | Okumoto et al. | Apr 2006 | A1 |
Number | Date | Country |
---|---|---|
0902370 | Mar 1999 | EP |
938046 | Aug 1999 | EP |
938046 | Aug 1999 | EP |
1 130 514 | Sep 2001 | EP |
1357476 | Oct 2003 | EP |
1357476 | Oct 2003 | EP |
62-274448 | Nov 1987 | JP |
02-037418 | Feb 1990 | JP |
07-191811 | Jul 1995 | JP |
7-244597 | Sep 1995 | JP |
09-288547 | Nov 1997 | JP |
10-508967 | Sep 1998 | JP |
10-283272 | Oct 1998 | JP |
11-065980 | Mar 1999 | JP |
2000-163329 | Jun 2000 | JP |
2000-242434 | Sep 2000 | JP |
2000-293317 | Oct 2000 | JP |
2001-067187 | Mar 2001 | JP |
2001-249853 | Sep 2001 | JP |
2001-331355 | Nov 2001 | JP |
2001-337850 | Dec 2001 | JP |
2002-157091 | May 2002 | JP |
2002-230246 | Aug 2002 | JP |
WO 9709676 | Mar 1997 | WO |
WO 03023640 | Mar 2003 | WO |
03027886 | Apr 2003 | WO |
03030431 | Apr 2003 | WO |
03030449 | Apr 2003 | WO |
Number | Date | Country | |
---|---|---|---|
20080016303 A1 | Jan 2008 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11016806 | Dec 2004 | US |
Child | 11773081 | US |