Patent Grant
7984227
The present invention relates to an art that connects a host computer and plural storage devices, and more particularly, to the control of various functions with which a storage device such a disk array provides the user.
In recent years, the intelligent formation of a computer storage device that is typically represented by a disk array is advancing. For example, a mirror function which a disk array device provides, when a host computer updates data of a storage area in the disk array device, automatically writes the update data even to another predetermined storage area. The storage data at that time is saved by instructing the disk array device to suspend the mirror function and can be used for an application such as backup. Because the writing of update data to another area is executed by a controller in the disk array device, a load is not imposed upon the host computer and high-speed input/output processing is enabled (for example, refer to U.S. Pat. No. 5,845,295).
On the other hand, the number of storage devices in a system increases and the storage devices of different types are intermixed. That is, the storage devices having different protocols are intermixed. At this point, “function” means capabilities to perform mirroring, take a snapshot, and so on provided by storage device, and “function control” means that control is performed so that a host computer can make the storage device implement these functions.
The host computer controls a function by accessing a function control interface which a disk array provides and sending an instruction according to a protocol unique to a predefined storage device. The function control interface can be realized in various forms such as an in-band SCSI command and TCP/IP communication via a LAN (for example, refer to U.S. Pat. No. 5,867,736).
Moreover, the spread of a storage area network (SAN) that connects plural host computers with plural storage devices through a network is advancing. The SAN is a high-speed network dedicated to data input/output that is typically a Fibre Channel and can improve the input/output performance of a computer. Further, the various utilization of storage areas is enabled, that is a computer group connected through the SAN can share one storage device, and, conversely, one computer can access plural storage devices. Recently, it has begun to introduce the art of storage virtualization that not only achieves sharing of a storage device/storage area but concatenates or partitions a storage area (disk volume) which a storage device provides and provides a host computer with a virtual storage area (virtual volume) to realize the flexible management of storage area.
In a computer system in which plural storage devices of different types are intermixed, such as SAN environment, when an intelligent function a storage device provides is utilized, a host computer must identify the type of a disk array to be controlled and send a function control instruction through an interface unique to each storage device. The control is complicated because even in the case to control similar functions, different control programs must be prepared for all different devices and used properly in the host computer.
Moreover, it is considered in the storage virtualization that intelligent functions work on virtual volumes are realized utilizing functions provided by storage devices. In this case, if a virtual volume has plural disk volumes, the function control must be instructed per disk volume. Accordingly, because a host computer must identify a device that provides the actual storage area of the virtual volume and use a control interface unique to each device, the control is complicated.
The present invention, in order to control various functions of a storage device of a different type, has been made in view of the above circumstances, and provides a common control interface that does not depend on the type of the device and implements the simplification of control.
The present invention installs a common interface for controlling the function of a storage device in a virtualization server and a switch of a computer system and provides a function control method that does not depend on the type of the storage device. A host computer sends a function control instruction to the common interface according to a predefined protocol regardless of the type of the storage device to be controlled. The device that provides the common interface manages the interrelationship between a storage area recognized by the host computer and a storage area provided by the storage device. Further, the device is provided with a means for converting the protocol of the common interface to a protocol unique to each device and a means for implementing the function control through an interface unique to each storage device.
When the device that provides the common interface receives a function control instruction, storage areas to which a function is applied and a storage device that provides the storage areas are interpreted. For example, if the instruction specifies a virtual volume, disk volumes that compose the virtual volume are interpreted. The type of the storage device that provides the storage area to be instructed is identified and function control is instructed through the function control interface unique to the device. The interface unique to each device is concealed by an interface server and the host computer can control a function using the common interface without being sensitive to the type of the device to be controlled. Because the host computer does not need to adapt to plural different interfaces and use each interface properly, the simplification of control can be achieved.
This embodiment, in storage virtualization environment in which a virtualization server virtualizes storage areas provided by two disk array devices connected through a Fibre Channel and having different function control interfaces, is an example of controlling a mirror function of a virtual volume realized using mirror functions provided by the disk array devices, utilizing a common interface which the virtualization server provides.
(1) System Configuration
There are two host computers 1101, 1201 and two disk arrays 1401, 1501, which are connected to Fibre Channel interfaces 1302, 1303, 1307, 1308 of a virtualization server 1301 through Fibre Channel interfaces 1102, 1202, 1402, 1502 and Fibre Channel cables 1314 to 1317 respectively. Data input/output from a host computer to a storage device is realized according to a standard SCSI protocol of these Fibre Channel cables. In the following description, the host computer and the disk array device are connected directly to the virtualization server. Because the present invention does not depend on a communication medium and a connection form, however, it can apply to even the connection via a switch and the configuration using another communication media and protocol such as a SCSI, not on a Fibre Channel network, and an IP network.
A virtual volume function control program 1106 for instructing virtual volume function control is provided in a memory 1104 of the host computer 1101 and executed by a CPU 1103. The virtual volume function program 1106 is installed previously in a host computer that sends a function control instruction of a disk array. The virtual volume function control program 1106, as described later, sends necessary information such as a controlling function and a virtual volume to be controlled to a common interface (virtual volume supplied as a common interface viewed from the host computer) provided by the virtualization server 1301 and instructs function control. The host computer 1201 also has the same configuration.
In the disk array 1401, a control program 1407 of a controller 1404 integrates the overall operation. The control program 1407 provides various intelligent functions such as a mirror function and a snapshot function in addition to an ordinary disk array function that provides disk volumes 1405 and 1406 as the storage area of the host computer. The disk array 1501 has also the same configuration as the disk array 1401 and provides the intelligent function that includes the mirror function.
In the virtualization server 1301, a volume virtualization program 1309 controls the overall operation and provides a storage virtualization function. Because the detailed implementation method of storage area virtualization is not the main purpose of the present invention, it is described briefly. When one virtual volume is created from plural disk volumes which disk arrays provide, the storage area of each disk volume is combined and made to appear like the one virtual volume. The address space of each disk volume is mapped to part of the address space of the virtual volume and a continuous address space is realized. The control program of the virtualization server 1301 manages the interrelationship between a virtual volume and actual disk volumes that compose the virtual volume and the interrelationship between the address space of the virtual volume and the address spaces of the actual disk volumes. The virtual volume is recognized as an LU (logical device) of a SCSI by a host computer, that is, an ordinary disk volume. When the host computer issues the Read or Write command to access the virtual volume, the virtualization server 1301 interprets the disk volume and address of a destination and issues a command to the actual disk volume. If the access destination is an area extending over plural disk volumes, plural commands are issued. Accordingly, one virtual volume that combines plural disk volumes can be realized.
Moreover, the virtualization server 1301 has a virtual volume management table 1313, a disk volume management table 1312, an interface correspondence table 1310, a function control program library 1311, and a common interface management table 1314 to realize a common interface for controlling a disk array function. These details will be described later.
(2) Description of Function Control of Disk Array Device
The function control of a disk array is described quoting a mirror function as an example. The mirror function handles one of two disk volumes as the primary volume and the other of them as the secondary volume and performs the duplication of data, thereby enabling five types of control that are initialization, suspension, re-synchronization, cancellation, and state acquisition.
For example, if the disk volume 1405 of the disk array 1401 is paired with the disk volume 1406 as the primary volume and the secondary volume, first, when mirror initialization is instructed, the contents of the primary volume 1405 are all copied to the secondary volume 1406. When the contents of the primary volume are updated after initialization is completed, the controller 1404 automatically updates even the contents of the corresponding secondary volume 1406. When mirror suspension is instructed, the disk volume 1406 is not updated automatically even if the data of the disk volume 1405 is updated, and the data at the time when mirror suspension is instructed remains in the disk volume 1406. When re-synchronization is instructed, the parts in which a difference occurred between the main and secondary volumes in the course of suspension are copied so that the two volumes can have the same contents. When cancellation is instructed, a pair of the main and secondary volumes is cancelled and the mirror function is nullified so long as initialization is not instructed again. When state acquisition is instructed, the controller 1404 returns the current state such as initialization being processed, being interrupted, or an error condition and so on. The state is returned through a control interface using a predefined data type in the same manner as a control instruction.
In the disk array 1401, a LAN interface 1403 is provided as the control interface of each function and externally receives an instruction according to a predetermined protocol. Based on the instruction, the controller 1404 performs function control. For example, the control block shown in
In the disk array 1501, there is no LAN interface, and a function control instruction is sent as the SCSI Mode Select command to a disk volume to be controlled through the Fibre Channel interface 1502. State acquisition is instructed with the Mode Sense command to the disk volume to be controlled and a state is returned as Sense data.
(3) Description of Common Interface
A common interface which a virtualization server provides can be realized by the same method as a function control interface a disk array provides. In this embodiment, however, an example of implementing the common interface as a virtual volume that is recognized as a disk volume connected through a Fibre Channel interface from a host computer is described.
The host computer recognizes the common interface as a readable/writable storage area in the same manner as other virtual volumes and exchanges control information by issuing the Read and Write commands. The disk volume that becomes an actual storage area is not allocated, however, and only an address space is provided by the volume virtualization program 1309. Function control is instructed by writing a control block having the data structure shown in
The volume virtualization program 1309, when the Write command to the first 512 bytes of a common interface is received, assumes the command to be a function control instruction and interprets the first 512 bytes in accordance with the data structure of
The administrator sets the WWN and LUN of the Fibre Channel interface allocated to the common interface as a parameter of the virtual volume function control program of the host computer. Otherwise, the virtual volume function control program of the host computer can automatically identify the common interface by issuing the Inquiry command to each LU connected to the Fibre Channel interface and searching the LU that returns a predefined vendor ID and a product ID unique to the common interface.
To realize the common interface, the virtualization server has the virtual volume management table 1313, the disk volume management table 1312, the interface correspondence table 1310, the function control program library 1311, and the common interface management table 1314. The common interface management table 1314 has the configuration shown in
The virtual volume management table 1313 has the configuration shown in
A disk volume is managed using the disk volume management table 1312 having the configuration shown in
Each API is provided by the function control program library 1311, and, for example, implemented as the function that is linked dynamically to the control program 1309 using the structure of
When a virtualization server recognizes a new disk volume, the administrator of the virtualization server creates each entry of the disk volume management table of
(4) Mirror Initialization Using Common Interface
An example of controlling the function of a disk array device using a common interface is described quoting the mirror initialization of a virtual volume.
As a preparation for describing, the function control using the common interface, the virtual volume creation procedure in this embodiment is described using the flowchart of
In
First, in step 7001, the Write command is issued from a host computer to a common interface and the control block of
Hereupon, whether the virtual volume to be controlled is processing other function control is checked with reference to a virtual volume management table (step 7012). If other function control is already being processed, process goes to step 7010. If other function control is not being processed, “being processed” is recorded in the control processing item of the virtual volume management table (step 7013). Further, a disk volume that constitutes each virtual volume is identified with reference to the virtual volume management table (step 7004). The example of the virtual volume management table of
Subsequently, in step 7005, disk volumes are combined mirror pairs. To combine the disk volumes into the mirror pairs, the primary volume and the secondary volume must exist in the same disk array device and volume capacities must be equal. Each volume capacity and the disk array device which provides the volume can be confirmed by referring the capacity and device identifier of a disk volume management table. Disk volumes which satisfy each of the above conditions are combined into mirror pairs which consist of a disk volume that becomes the primary volume and a disk volume that becomes the secondary volume. If there is a pair of which capacity does not match or a pair that does not exist in the same device, process terminates with a mirror initialization failure (step 7006). If there are plural combinations of pairs that satisfy the conditions, for example, the virtual volumes that become the primary volume and the secondary volume have the disk volumes 1, 3 and 2, 4 respectively, and a pair can be combined with the disk volumes 1, 2 and 3, 4 as a pair. On the other hand, the case where the pair can be formed with the disk volumes 1, 4 and 3, 2 is also possible. In such a case, the smallest disk volume identifiers are combined. In the case of the aforementioned example, the disk volume that becomes the primary volume is 1, 3 in the ascending order of an identifier and the disk volume that becomes the secondary volume is 2, 4 in the same manner. Accordingly, 1 and 2, and 3 and 4 are paired in the ascending order of the identifier. Consequently, even when the pair relationship of disk volumes must be known for another function instruction, disk volume pairs which are identical to what are made in an initialization process are always obtained for a virtual volume.
After it is confirmed that a mirror can be formed, mirror initialization instructions are sent to disk array devices for all pairs. The control data that becomes an argument which has the structure of
A mirror initialization instruction is issued in this manner which includes converting a control block received from a host computer through a common interface in step 7002 to a protocol unique to a disk volume in steps 7007 and 7008. According to the aforementioned procedure, the mirror function of a virtual volume that has plural disk volumes can be controlled from the common interface regardless of the type of a disk array device.
(5) Mirror State Acquisition Using Common Interface
Even when a mirror state of a virtual volume is acquired, an instruction is sent from a host computer to a common interface in the similar procedure to the aforementioned mirror initialization. A virtualization server calls an API for acquiring the mirror state from a disk array device. In the case of state acquisition, however, the difference is that the virtualization server must convert the mirror states of the disk volumes acquired from disk array devices to a mirror state of a virtual volume and transfer it to the host computer.
The mirror state acquisition procedure is described using the flowchart of
After a mirror pair state of a virtual volume is created, the record of “being processed” of a virtual volume management table is deleted (step 10012) and a status is returned to a host computer (step 10013). This status is defined in a standard as part of the SCSI protocol. Usually, if there is no error, Good (byte code=00000) is returned. Accordingly, the virtual volume function control program of the host computer recognizes the completion of state acquisition operation and sends the Read command to a common interface to read a state. The volume virtualization program of a virtualization server returns the mirror pair state of the virtual volume created in the step 1001 as the data for the Read command (step 10015) and finally returns a status (status 10016).
As described above,
According to the aforementioned procedure, function control in which information must be transferred from a virtualization server to a host computer can also be realized.
This embodiment, in storage virtualization environment in which plural disk array devices having a different function control interface exist, can apply a storage function such as a mirror function, to virtual volumes. At that time, instruction of function control can be made by a common method without being sensitive to which device a constituting storage area exists in.
In this embodiment, the basic function of storage virtualization and the common interface of the function control are provided by the same virtualization server, but each function can also be realized in an individual device. Further, this embodiment can also apply to a switch device having the same storage virtualization function as well as the server. Here, these devices are called connecting devices of storage devices. Furthermore, this embodiment can also apply to the function control in a unit of a file system, for example, by replacing a disk array with a file server, a Fibre Channel and the SCSI protocol with a LAN and the NFS protocol, a management unit of a virtualization server with a file system, and a common interface with a LAN interface.
The configuration of this embodiment is almost the same as the first embodiment of
(1) Plural Function Control Interfaces
In this embodiment, a virtualization server provides a host computer with two common interfaces. In addition to an interface by a virtual volume described in the first embodiment, an interface through a LAN using the LAN interface 1306 of
The virtual volume function control program 1206 of the host computer 1201 of
The disk array 1401 provides a function control method by the Mode Select/Mode Sense command via a Fibre Channel interface in the same manner as the disk array 1501 in addition to the function control through the LAN interface described in the first embodiment. That is, the volume virtualization program of a virtualization server can send a function control instruction to the disk array 1401 both via the LAN and via the Fibre Channel.
The volume management table of a virtualization server is modified to correspond to plural control interfaces of the disk array 1401. Further, an item is added to define recovery processes for errors that occur while a function control is processed. The modified disk volume management table is shown in
A recovery API 14109 is called when error handling is performed. For example, the virtualization server is instructed a function control for a virtual volume having two disk volumes. In this case, when the control for one disk volume is successful and the control for the other disk volume is unsuccessful, this API is used to cancel the successful instruction, in order to return the state of the virtual volume to the state before the instruction is issued. A Recovery API is called with the same argument as the API and registered in the same method as the API.
(2) Error Handling
The function control instruction procedure in this embodiment is almost the same as the first embodiment, but the points that differ are that error handling is performed using plural control interfaces which a disk array provides and the Recovery API previously registered in a volume management table. Only the modification part from the first embodiment is described regarding the mirror initialization instruction procedure. The API of mirror cancellation is assumed to have been registered previously in the Recovery API of the mirror initialization of the volume management table.
In this embodiment, plural common interfaces provided by a virtualization server allow host computers select communication means. Accordingly, even when each host computer has different communication means to send a control instruction, the same effect as the first embodiment is obtained. Further, when a function for a virtual volume having plural disk volumes is controlled, plural control instructions are sent to a disk array. When some instructions fail instructions completed successfully are cancelled and the virtual volume can be returned to a consistent state. In this embodiment, a single Recovery API was defined for each API, but the Recovery APIs used for various error condition can also be defined. In other words, in a third embodiment, as shown in
This embodiment is an example of applying the present invention to SAN environment in which a Fibre Channel network is used and utilizing a common interface for the function control of disk volumes. The common interface is provided by a Fibre Channel switch instead of a virtualization server. Further, a security function that records a function control log at a management console and restricts the control that can be sent from a host computer is provided. In this embodiment, instructions of function control are the Mode Select and Mode Sense commands sent to a disk volume to be controlled and the common interface is also the volume.
(1) System Configuration
The configuration and operation of host computers 16101, 16201 and disk arrays 16401, 16501 are the same as the first embodiment, and their devices are connected to a Fibre Channel switch 16301. The virtual volume function control program of the host computer sends the Mode Select and Mode Sense commands instead of the Read and Write commands unlike the first embodiment. Each page code uses a vendor unique number defined previously for function control. Data structure of
Regarding the Fibre Channel switch 16301, the points that differ from a usual Fibre Channel switch is described. A port module 16302 that provides a Fibre Channel port connected to a host computer or a disk array consist of with a Fibre Channel interface 16310 and an interface control processor 16311. When the interface control processor 16311 receives a control instruction for a common interface, the instruction is transmitted to a CPU 16307. Specifically, if a CDB (command descriptor block) of the SCSI command received by a Fibre Channel interface is interpreted and the received command is the Mode Select or Mode Sense command having a page code defined for function control, it is determined that the command is sent to access to the common interface, and the WWN of a host computer which is a source of the command, the WWN and LUN of a destination volume, and the contents of a command are posted to the CPU 16307.
The CPU 16307 executes a switch control program 16312 in a memory 16308 and controls the switch operation. An interface correspondence table 16313, a volume control program library 16314, and a volume management table 16315 have the same configuration of the first embodiment. The switch control program 16312 converts a protocol using these tables. Because the details are the same as the first and second embodiments, they are omitted. Because this embodiment does not have a storage virtualization function, a virtual volume management table is not provided. Further, because a disk volume itself to be controlled becomes a destination of a control command from a host computer, that is, a common interface, this embodiment does not have a common interface management table.
A control authority management table 16316 is constructed as shown in
When the switch control program 16312 receives a control instruction through a common interface, a record of control is transmitted from a LAN interface 16309 to a management console 16601 specified with an IP address and port number which are set previously by the administrator. The contents of the record include time of a control source instruction, whether the control is permitted or not object to be controlled, contents of control, an interface used to send an instruction to a disk array, and information that identifies the success or failure of control.
In the management console 16601, a log record program 16605 which logs a control record is executed by a CPU 16602. The log record program waits for the arrival of a control record from a Fibre Channel switch at a LAN interface 16603 and records the result in a log file 16606. Further, the log record program has a function that displays a log so that the administrator can browse recorded information.
(2) Restriction on Sending Control Instruction to Common Interface and Recording of Log
In this embodiment, the operation of a switch control program 16312 that sends the Mode Select command for the function control of a disk volume is described using the flowchart of
First, the virtual volume control program of a host computer sends the Mode Select command to a disk volume to be controlled (step 19001). The control block shown in
If the source WWN of an instruction is not recorded in the list of WWN that have permission, the control is not permitted. Accordingly, a record indicating that an instruction is rejected is sent to the log record program of a management console from a LAN interface (step 19004) and processing proceeds to step 19010. On the other hand, if the control is permitted, instruction of function control is sent to a disk array using the API and control interface registered in the disk volume management table in the same manner as the first embodiment (step 19006). The success or failure of an API call that is the function control of a disk array is examined (step 19007) and the success or failure of the API call is recorded in a log in steps 19008, 19009 respectively. Finally, the SCSI status is returned to the source host computer of an instruction (step 19010) and operation finishes.
According to this embodiment, in usual SAN environment in which storage virtualization is not used, the function control of disk volumes using a common interface is enabled. Further, it is possible to manage authority of function control by specifying host computers that have permission for each function of each disk volume. Furthermore, a history of control can be recorded in a log. This embodiment manages the control authority in a unit of a host computer, but other management units can be applied as well. For example, authority can be managed in a unit of a user by managing control permission for user in a control authority management table and including a user ID in the control instruction from the host computer. Further, in this embodiment, log information is sent immediately to an external management console, but it can also be logged in the internal storage area of a switch and sent to the management console or acquired from the management console periodically.
As described above, according to the embodiments of the present invention, an interface unique to each device is concealed. A host computer can control a function using a common interface without being sensitive to the type of the device to be controlled. Accordingly, the configuration of a program that issues a control instruction is simplified.
By providing plural common interfaces a communication means a host computer can select. Accordingly, the present invention can even apply to environments in which devices having different communication means that can be used for sending a control instruction are intermixed. Further, when a series of types of function control for plural disk volumes are ordered, the function control of error handling can be defined for the event in which the part of control fails.
Moreover, permission of function control to computers can be managed for each type of function control for each object to be controlled. Further, a control history can be recorded in a log.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2002-356477 | Dec 2002 | JP | national |
The present application is a continuation of application Ser. No. 11/281,475, filed Nov. 18, 2005 now U.S. Pat. No. 7,392,336; which is a continuation of application Ser. No. 10/372,913, filed Feb. 26, 2003, now U.S. Pat. No. 7,020,734, the contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4805090 | Coogan | Feb 1989 | A |
| 5249279 | Schmenk et al. | Sep 1993 | A |
| 5333277 | Searls | Jul 1994 | A |
| 5357475 | Hasbun et al. | Oct 1994 | A |
| 5659704 | Burkes et al. | Aug 1997 | A |
| 5761705 | DeKoning et al. | Jun 1998 | A |
| 5794013 | McBrearty | Aug 1998 | A |
| 5809285 | Hilland | Sep 1998 | A |
| 5845295 | Houseman et al. | Dec 1998 | A |
| 5867736 | Jantz | Feb 1999 | A |
| 5881311 | Woods | Mar 1999 | A |
| 5941972 | Hoese et al. | Aug 1999 | A |
| 6098128 | Velez-McCaskey et al. | Aug 2000 | A |
| 6105122 | Muller et al. | Aug 2000 | A |
| 6230240 | Shrader et al. | May 2001 | B1 |
| 6321298 | Hubis | Nov 2001 | B1 |
| 6430645 | Basham | Aug 2002 | B1 |
| 6535954 | Obara et al. | Mar 2003 | B2 |
| 6542954 | Aruga | Apr 2003 | B1 |
| 6553408 | Merrell et al. | Apr 2003 | B1 |
| 6625691 | Obara et al. | Sep 2003 | B2 |
| 6654830 | Taylor et al. | Nov 2003 | B1 |
| 6683883 | Czeiger et al. | Jan 2004 | B1 |
| 6834324 | Wood | Dec 2004 | B1 |
| 6862648 | Yatziv | Mar 2005 | B2 |
| 6877044 | Lo et al. | Apr 2005 | B2 |
| 6889309 | Oliveira et al. | May 2005 | B1 |
| 6934878 | Massa et al. | Aug 2005 | B2 |
| 6965956 | Herz et al. | Nov 2005 | B1 |
| 6973516 | Athanas et al. | Dec 2005 | B1 |
| 7020734 | Mimatsu et al. | Mar 2006 | B2 |
| 7366853 | Honda et al. | Apr 2008 | B2 |
| 7392336 | Mimatsu et al. | Jun 2008 | B2 |
| 7433948 | Edsall et al. | Oct 2008 | B2 |
| 7548975 | Kumar et al. | Jun 2009 | B2 |
| 20020095549 | Matsunami et al. | Jul 2002 | A1 |
| 20020109584 | Sakaki | Aug 2002 | A1 |
| 20020172195 | Pekkala et al. | Nov 2002 | A1 |
| 20030093439 | Mogi et al. | May 2003 | A1 |
| 20030126327 | Pesola et al. | Jul 2003 | A1 |
| 20030204572 | Mannen et al. | Oct 2003 | A1 |
| 20030204597 | Arakawa et al. | Oct 2003 | A1 |
| 20040019706 | Smith | Jan 2004 | A1 |
| 20040117522 | Loffink et al. | Jun 2004 | A1 |
| 20050036499 | Dutt et al. | Feb 2005 | A1 |
| 20050066078 | Yagisawa et al. | Mar 2005 | A1 |
| 20050138258 | Seto | Jun 2005 | A1 |
| 20050149656 | Seto | Jul 2005 | A1 |
| 20060004935 | Seto et al. | Jan 2006 | A1 |
| 20060036786 | Kreiner et al. | Feb 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 10145451 | May 1998 | JP |
| 11338648 | Dec 1999 | JP |
| 2000284982 | Oct 2000 | JP |
| 2001325207 | Nov 2001 | JP |
| 2002244817 | Aug 2002 | JP |
| 2002247104 | Aug 2002 | JP |
| 2002312251 | Oct 2002 | JP |
| 0101257 | Jan 2001 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20080222356 A1 | Sep 2008 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11281475 | Nov 2005 | US |
| Child | 12118866 | US | |
| Parent | 10372913 | Feb 2003 | US |
| Child | 11281475 | US |
