STORAGE SYSTEM AND METHOD FOR CONTROLLING STORAGE SYSTEM

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2012-147887, filed on Jun. 29, 2012, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a storage system and a method for controlling a storage system.

BACKGROUND

In a storage system, a controller module (hereinafter, referred to as a CM) performs data control for a plurality of disk devices. For example, the CM receives an input/output (I/O) command to a disk device from a host computer via a channel adapter (hereinafter, referred to as a CA), which is an interface to the host computer. Then, the CM controls output and input of data to and from the disk device on the basis of the received I/O command.

Some CAs have a dump function of recording an internal file and the memory contents as dump data (fault information). When an error occurs in a CA having a dump function, a central processing unit (CPU) in a CM collects dump data from the CA in which the error has occurred and saves the collected dump data into a storage device called a bootup and utility device (BUD) in a system.

The CA in which the error has occurred is isolated from the CM. The dump data saved in the BUD includes fault information and is used for analysis of the error in the CA and recovery from the error.

Japanese Laid-open Patent Publication No. 2006-107053, Japanese Laid-open Patent Publication No. 2001-34508, Japanese Laid-open Patent Publication No. 2007-334668, and Japanese Laid-open Patent Publication No. 2003-330781 disclose related techniques.

However, in the related arts described above, there are some cases in which recording of dump data is not performed. For example, when an error has occurred in the BUD in succession to an error in a CA, recording of dump data is not performed.

SUMMARY

According to an aspect of the present invention, provided is a storage system including a plurality of control devices to control output and input of data to and from a storage device. At least one of the plurality of control devices includes an interface unit and an arithmetic processing unit. The interface unit receives an instruction regarding output or input of data to or from the storage device. The arithmetic processing unit receives the instruction from the interface unit and executes the instruction. The arithmetic processing unit selects, when an error has occurred in a specific interface unit, a plurality of processing units and requests the selected plurality of processing units to execute saving processing for saving dump data of the specific interface unit. The specific interface unit is included in one of the plurality of control devices. The plurality of processing units are included in respective control devices out of the plurality of control devices.

The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a functional block diagram illustrating a configuration of a storage system according to a first embodiment;

FIG. 2 is a functional block diagram illustrating a configuration of a storage system according to a second embodiment;

FIG. 3 illustrates an example of a data structure of a siphoning control table used in a second embodiment;

FIG. 4A illustrates an example of a processing operation of saving processing for dump data;

FIG. 4B illustrates an example of a processing operation of saving processing for dump data;

FIG. 4C illustrates an example of a processing operation of saving processing for dump data;

FIG. 5A illustrates an example of dump data that is saved into a BUD of each CM after a degradation of CM#5 occurs in serving as a saving destination of CA dump and CM#7 is newly selected as a saving destination;

FIG. 5B illustrates an example of processing for copying dump data from a BUD of CM#6 to a BUD of CM#7;

FIG. 5C illustrates an example of dump data that is saved into a BUD of CM#7 after dump data is copied from a BUD of CM#6;

FIG. 5D illustrates an example of the case where after a degradation of CM#5 occurs in serving as a saving destination of CA dump, a degradation of CM#6 also occurs serving as another saving destination of CA dump; and

FIG. 5E illustrates an example of dump data that is saved into a BUD of CM#7 in the case where a degradation of CM#6 also occurs after a degradation of CM#5 occurs.

FIG. 6A is a sequence diagram illustrating siphoning processing for CA dump by a storage system;

FIG. 6B is a sequence diagram illustrating siphoning processing for CA dump by a storage system;

FIG. 7A is a flowchart illustrating a processing procedure for designating a siphoning destination in the second embodiment; and

FIG. 7B is a flowchart illustrating the processing procedure of the processing for designating a siphoning destination in the second embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a storage system and a method for controlling a storage system according to embodiments will be described in detail with reference to the drawings. Embodiments are not limited by the described embodiments. The individual embodiments may be combined in an appropriate manner as long as no contradiction arises in the processing details.

First Embodiment
Configuration of Storage System According to First Embodiment

FIG. 1 is a functional block diagram illustrating a configuration of a storage system 2 according to a first embodiment. As illustrated in FIG. 1, the storage system 2 according to the first embodiment includes a controller module (hereinafter, referred to as a CM) 3, which is a control device, disks 4, a front-end router (hereinafter, referred to as an FRT) 5. The storage system 2 according to the first embodiment also includes a back-end router (hereinafter, referred to as a BRT) 6 and a drive enclosure (hereinafter, referred to as a DE) 7. The number of CMs 3 provided in the storage system 2 is not limited to the example illustrated in FIG. 1 and three or more CMs 3 are provided in the storage system. Furthermore, the number of disks 4 provided in the storage system 2 is not limited to the example illustrated in FIG. 1.

The storage system 2 according to the first embodiment is connected to host computers 1 that are information processing apparatuses serving as host devices. The storage system 2 receives an I/O command to a disk 4 from a host computer 1, and controls the received I/O command. The number of the host computers 1 connected to the storage system 2 is not limited to the example illustrated in FIG. 1.

The CM 3 is a device that controls output and input of data to and from the disk 4. In the example illustrated in FIG. 1, the number of CMs 3 is three. In the case where the CMs 3 are distinguished from one another, each CM 3 is described as a CM#0, a CM#1, or a CM#2, appropriately, as illustrated in FIG. 1.

The disk 4 is a storage device that stores data. The disk 4 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like. The FRT 5 allows connection between the CMs 3. The FRT 5 includes four paths used for communication between the CMs 3. The BRT 6 allows connection between one of the CMs 3 and one of the disks 4. The DE 7 is a casing to which the disks 4 are loaded.

Functional Configuration of CM

The CM 3 includes a plurality of channel adapters (hereinafter, referred to as CAs) 31, which is an interface unit for connecting to the host computers 1, and an I/O controller (hereinafter, referred to as an IOC) 32. The CM 3 also includes a BUD 33, a main control unit 34, a sub-control unit 35, an inter-CM communication driver 36, and a storage unit 37. In the case where the main control unit 34 and the sub-control unit 35 are not distinguished from each other, the main control unit 34 and the sub-control unit 35 are referred to as control units. A control unit is an arithmetic processing unit. A single path is used for communication between a CA and a CM in which the CA exists. The number of control units provided in the CM 3 is not limited to the example illustrated in FIG. 1. For example, the CM 3 may include a single control unit.

A CA 31 is a communication interface for communicably connecting to the host computer 1. For example, the CA 31 receives an I/O command, which is a command regarding input and output of data stored in the disk 4, from the host computer 1. A plurality of CAs 31 exist in the CM 3. In the example illustrated in FIG. 1, four CAs 31 exist. In the case where the CAs 31 are distinguished from one another, the CAs 31 are referred to as a CA 31-1 (CA#1), a CA 31-2 (CA#2), a CA 31-3 (CA#3), and a CA 31-4 (CA#4).

The CA 31 has a dump function of recording an internal file and memory contents as dump data. In the case where an error has occurred in a CA 31, dump data stored in the CA 31 is collected by the main control unit 34 and is saved into the BUD 33 in the storage system 2. In the description provided below, dump data stored in the CA 31 is referred to as “CA dump”. Processing of separating a CM 3, a CA 31 in a CM 3, or another unit in a CM 3 is referred to as “degradation processing”, which is performed when an error occurs in the CM 3, the CA 31, or the other unit.

The IOC 32 is a communication interface for communicably connecting to the disks 4. The BUD 33 is a storage device which stores, in the case where an error has occurred in a CA 31, dump data collected from the CA 31 by the main control unit 34. The dump data saved in the BUD 33 is used later for analyzing the error in the CA 31.

The main control unit 34 is connected to the CAs 31 and is a main control unit for the case where processing in the CM 3 is performed in a distributed manner. Hereinafter, description will be provided on the assumption that the main control unit 34 is a CPU. However, the main control unit 34 may be an electronic circuit such as a micro-processing unit (MPU). The main control unit 34 functions as various functional parts by, for example, executing controller module firmware (CFW) for controlling the CM 3.

In the case where an error has occurred in a CA 31 to which the main control unit 34 is connected, the main control unit 34 stores dump data collected from the CA 31 in the storage unit 37, compresses the dump data in the storage unit 37, and saves the compressed dump data into the BUD 33. Processing for collecting dump data from the CA 31 and then saving the collected dump data into the BUD 33 is referred to as “siphoning processing”.

The main control unit 34 selects a main control unit 34 or a sub-control unit 35 provided in another CM 3, and requests the selected main control unit 34 or the selected sub-control unit 35 to perform saving processing for dump data.

For example, in the case where an error has occurred in a CA 31 of the CM#0, the main control unit 34 of the CM#0 saves dump data collected from the CA 31 into the BUD 33 of the CM#0. Furthermore, the main control unit 34 of the CM#0 requests the main control unit 34 of the CM#1 to perform processing for saving the dump data collected from the CA 31 of the CM#0. Accordingly, dump data collected from the CA 31 of the CM#0 is saved also into the BUD 33 of the CM#1.

Furthermore, the main control unit 34 newly selects, in the case where an error has occurred in a main control unit 34 or a sub-control unit 35 of a selected CM 3, another main control unit 34 or another sub-control unit 35 and requests the newly selected main control unit 34 or the newly selected sub-control unit 35 to perform saving processing for dump data of the CA 31.

For example, in the case where an error has occurred in the CM#1, the main control unit 34 of the CM#0 newly selects the main control unit 34 of the CM#2 and requests the newly selected main control unit 34 of the CM#2 to perform saving processing for dump data collected from the CA 31 of the CM#0.

The sub-control unit 35 is connected to the CAs 31 and is a sub-control unit for the case where processing in the CM 3 is performed in a distributed manner. Hereinafter, description will be provided on the assumption that the sub-control unit 35 is a CPU. However, the sub-control unit 35 may be an electronic circuit such as an MPU. For example, the sub-control unit 35 functions as various functional parts by executing CFW for controlling the CM 3.

The inter-CM communication driver 36 is a communication interface for communicably connecting to other CM 3 via the FRT 5. The storage unit 37 stores various data to be used for processing by the main control unit 34 or the sub-control unit 35.

As described above, in the storage system 2 according to the first embodiment, the CM 3 duplicates a saving destination of dump data collected from the CA 31 provided in the CM 3. Thus, even in the case where an error has occurred in the BUD 33 of the CM 3, the dump data collected from the CA 31 may be saved. Furthermore, the CM 3 newly selects, in the case where an error has occurred in a main control unit 34 or a sub-control unit 35 of a selected CM 3, another main control unit 34 or another sub-control unit 35 and requests the newly selected main control unit 34 or the newly selected sub-control unit 35 to perform saving processing for the dump data of the CA 31. Thus, even in the case where an error has occurred in the duplicated saving destination, the dump data collected from the CA 31 may be saved.

Second Embodiment

In the first embodiment, the case where a main control unit 34 connected to a CA 31 in which an error has occurred performs processing for siphoning dump data and requests another control unit to perform processing for siphoning dump data. In a main control unit 34 connected to a CA in which an error has occurred, the I/O load may be high. In this case, the main control unit 34 may not be able to save all the collected data into the BUD 33. Thus, in the case where an error has occurred in a CA, a plurality of main control units 34 or sub-control units 35 each having a low I/O load may be selected from the storage system, and the selected plurality of main control units 34 or sub-control units 35 may be caused to perform saving processing.

In a second embodiment, an example in which a master CM set in the storage system selects, in the case where an error has occurred in a CA, a plurality of main control units 34 or sub-control units 35 each having a low I/O load and causes the selected plurality of main control units 34 or sub-control units 35 to perform saving processing will be described.

Configuration of Storage System According to Second Embodiment

FIG. 2 is a functional block diagram illustrating a configuration of a storage system 2A according to the second embodiment. As illustrated in FIG. 2, the storage system 2A includes a CM 3A, CMs 3B, disks 4, and an FRT 5, a BRT 6, and a DE 7. The storage system 2A is connected to host computers 1 serving as host devices. In the storage system 2A according to the second embodiment, units having functions similar to those in the configuration of the storage system 2 illustrated in FIG. 1 are referred to with the same reference numerals and the detailed description thereof will be omitted. In the example illustrated in FIG. 2, regarding the number of CMs provided in the storage system 2A, eight CMs in total, that is, one CM 3A and seven CMs 3B, are provided in the storage system 2A. However, the number of CMs provided in the storage system 2A is not limited to the example illustrated in FIG. 2 as long as three or more CMs are provided in the storage system 2A. In FIG. 2, only four of the seven CMs 3B are illustrated. The number of disks 4 provided in the storage system 2A is not limited to the example illustrated in FIG. 2.

Here, for example, the CM 3A is defined as a master CM and the CMs 3B are defined as slave CMs. The master CM has a function of selecting, within the storage system 2A, a plurality of main control units or sub-control units to which execution of saving processing is requested in the case where an error has occurred in a CA. For the convenience of description, the CM 3A is described as a CM#0 when appropriate. Furthermore, in the case where the CMs 3B are distinguished from one another, the CMs 3B are described as a CM#1, a CM#5, a CM#6, and a CM#7, as illustrated in FIG. 2.

Functional Configuration of Master CM

The CM 3A is a device that controls output and input of data to and from the disks 4. The CM 3A includes a plurality of CAs 31, an IOC 32, a BUD 33, a main control unit 34A, a sub-control unit 35A, an inter-CM communication driver 36, and a storage unit 37A. In the CM 3A in the second embodiment, units having functions similar to those in the configuration of the CM 3 illustrated in FIG. 1 are referred to with the same reference numerals and the detailed description thereof will be omitted. The number of control units provided in the CM 3A is not limited to the example illustrated in FIG. 2. For example, the CM 3A may include a single control unit.

The main control unit 34A is a main control unit for the case where processing in the CM 3A is performed in a distributed manner. The main control unit 34A includes a siphoning processing part 51, a siphoning destination designating part 52, and an information acquiring part 53. Hereinafter, description is provided on the assumption that the main control unit 34A is a CPU. However, the main control unit 34A may be an electronic circuit such as an MPU. The main control unit 34A functions as various functional parts by, for example, executing CFW for controlling the CM 3A.

Upon detecting an error in a CA 31 connected to a control unit, the siphoning processing part 51 included in the control unit causes the siphoning destination designating part 52 of the master CM 3A to designate a siphoning destination for dump data of the CA 31 in which the error is detected. For example, the siphoning processing part 51 transmits, via the inter-CM communication driver 36, a request for acquiring a siphoning destination to the siphoning destination designating part 52 of the master CM 3A. At this time, the siphoning processing part 51 transfers the size of the dump data to the siphoning destination designating part 52 of the master CM 3A.

Then, the siphoning processing part 51 requests the siphoning destination, which is designated by the siphoning destination designating part 52 of the master CM 3A, to save the dump data of the CA 31 in which the error is detected.

The siphoning processing part 51 performs, when requested from another siphoning processing part 51 to save dump data, saving processing for dump data for the CA 31 in which an error has occurred. For example, the siphoning processing part 51 acquires dump data of the CA 31 in which an error has occurred, and saves the acquired dump data into the storage unit 37A with which the siphoning processing part 51 is connected. Then, the siphoning processing part 51 compresses the dump data stored in the storage unit 37A, and saves the compressed dump data into the BUD 33.

Furthermore, in the case where an error has occurred in any one of control units requested to perform saving processing, the siphoning processing part 51 causes the siphoning destination designating part 52 of the master CM 3A to designate a new siphoning destination of the dump data of the CA 31. Then, the siphoning processing part 51 requests the new siphoning destination designated by the siphoning destination designating part 52 of the master CM 3A to perform siphoning of dump data of the CA 31 in which the error is detected.

Furthermore, in the case where an error has occurred in any one of control units requested to perform saving processing, the siphoning processing part 51 causes a newly selected control unit to save dump data that has been saved by a normal control unit and that has not been saved by the newly selected control unit. Processing for saving dump data that has been saved by the normal control unit and that has not been saved by the newly selected control unit is referred to as “copying processing”.

When a request for designating a siphoning destination is issued from the siphoning processing part 51 of the main control unit 34A, the main control unit 34B, or the sub-control unit 35A, the siphoning destination designating part 52 performs the processing described below. That is, the siphoning destination designating part 52 selects a plurality of control units to be requested to perform saving processing from among control units whose dump data saving processing time is shorter than a specific time and whose I/O processing time, which is calculated on the basis of the number of I/O commands, is shorter.

For example, the siphoning destination designating part 52 narrows candidates for a siphoning destination down to control units whose siphoning time for dump data of the CA 31 in which the error is detected is shorter than a specific time. Furthermore, the siphoning destination designating part 52 designates, for siphoning destinations, two control units having processing times, which is converted from the number of I/O commands being processed, shorter than processing times of other candidates.

Processing performed by the siphoning destination designating part 52 for estimating whether or not processing for siphoning dump data will be completed within a specified time with certainty will be described. For example, the siphoning destination designating part 52 calculates an estimated time for dump data siphoning processing as a “CA dump siphoning time” for each control unit on the basis of the size of CA dump data, and stores the calculated estimated time into a siphoning control table 371. The siphoning destination designating part 52 calculates the “CA dump siphoning time” on the basis of Equation (1). Equation (1) is as follows:

CA dump siphoning time={(CM/CA communication time)+(CM/CM communication time)+(data compression time)+(BUD saving time)}×(data size) (1)

Here, the data size represents a data size of dump data of a CA in which an error has occurred. The CM/CA communication time represents communication time for the case where data of 1 megabyte (MB) is communicated between a CA in which an error has occurred and a CM including the CA. The CM/CM communication time represents communication time for the case where data of 1 MB is transferred from a CM including a CA in which an error has occurred to a CM of a siphoning destination, and is used for the case where the siphoning destination is located outside the CM including the CA in which the error has occurred. The data compression time represents the time for compressing data of 1 MB. The BUD saving time represents the time for saving compressed data of 1 MB.

CA dump siphoning execution time=(CA dump siphoning time)/{1−(usage rate of control unit)} (2)

Then, the siphoning destination designating part 52 stores the calculated “CA dump siphoning execution time” for each control unit into the siphoning control table 371. The siphoning destination designating part 52 compares the “CA dump siphoning execution time” for each control unit stored in the siphoning control table 371 with a specified time allowed for siphoning, and narrows candidates for a siphoning destination down to control units that are capable of completing siphoning within the specified time. Here, the siphoning destination designating part 52 selects, as candidates for a siphoning destination, control units for which “active flag” is not “ON”.

Next, processing performed by the siphoning destination designating part 52 for designating a control unit whose input/output processing time is the shortest among candidates for a siphoning destination will now be described. For example, the siphoning destination designating part 52 acquires the number of I/O commands being processed by each control unit on the basis of command number information 373 for each CM. Then, the siphoning destination designating part 52 calculates the “I/O processing time” for each control unit, which is the processing time for each control unit converted from the acquired number of I/O commands. The siphoning destination designating part 52 calculates the “I/O processing time” of each control unit, using Equation (3), on the basis of the number of I/O commands for straight access not using inter-CM communication and the number of I/O commands for cross access using inter-CM communication. Equation (3) is as follows:

I/O processing time=(the number of I/O commands for cross access)×(I/O command processing time for cross access)+(the number of I/O commands for straight access)×(I/O command processing time for straight access) (3)

The number of I/O commands for cross access represents the number of I/O commands being processed for cross access. The number of I/O commands for cross access includes the number of I/O commands received by inter-CM communication as well as the number of I/O commands transmitted by inter-CM communication. The I/O command processing time for cross access represents the processing time for the case of processing of one I/O command for cross access. The number of I/O commands for straight access represents the number of I/O commands being processed for straight access. The I/O command processing time for straight access represents the processing time for the case of processing of one I/O command for straight access.

Then, the siphoning destination designating part 52 stores the calculated I/O processing time for each control unit into the siphoning control table 371. Then, the siphoning destination designating part 52 designates two siphoning destinations, from among the narrowed down candidates for a siphoning destination, in order from the control unit whose I/O processing time for the control unit stored in the siphoning control table 371 is the shortest. The siphoning destination designating part 52 notifies the requesting CM of the designated siphoning destination. Accordingly, the requesting CM is capable of requesting the designated siphoning destination to collect and save dump data of a CA 31 in which an error is detected.

There may be no control unit whose siphoning time for dump data of a CA 31 in which an error is detected is shorter than a specified time allowed for siphoning dump data. In such a case, the siphoning destination designating part 52 may designate a control unit whose CA dump siphoning execution time is the shortest as a siphoning destination. Accordingly, the siphoning destination designating part 52 is capable of causing the designated siphoning destination to collect and save the dump data for the specified time and to perform an examination regarding error using the collected and saved dump data.

Furthermore, in the case where an error has occurred in any one of selected control units, the siphoning destination designating part 52 newly selects another control unit and requests the newly selected control unit to perform saving processing for dump data of the CA 31. Here, the siphoning destination designating part 52 narrows candidates for a siphoning destination down to control units whose siphoning time for dump data of the CA 31 in which the error has been detected is shorter than a specific time. Furthermore, the siphoning destination designating part 52 designates, as a new siphoning destination, a control unit whose processing time converted from the number of I/O commands currently being processed is the shortest among the narrowed down candidates for a siphoning destination.

The information acquiring part 53 acquires a value of the usage rate of the control unit, and includes the acquired value into the CPU usage rate information 372 stored in the storage unit 37A, which will be described later. The information acquiring part 53 acquires a value of the number of I/O commands, and includes the acquired value into the command number information 373 stored in the storage unit 37A, which will be described later.

The sub-control unit 35A is a sub-control unit for the case where processing in the CM 3A is performed in a distributed manner. The sub-control unit 35A includes a siphoning processing part 51 and an information acquiring part 53. Hereinafter, description will be provided on the assumption that the sub-control unit 35A is a CPU. However, the sub-control unit 35A may be an electronic circuit such as an MPU. The sub-control unit 35A functions as various functional parts by, for example, executing CFW for controlling the CM 3A.

The storage unit 37A stores various data to be used for processing by the main control unit 34A or the sub-control unit 35A. For example, the storage unit 37A stores the siphoning control table 371, the CPU usage rate information 372, and the command number information 373.

The siphoning control table 371 stores, for each control unit, the estimated time for siphoning processing and the processing time converted from the number of input/output commands being processed in association with each control unit. The siphoning control table 371 is created by, for example, the siphoning destination designating part 52, and is used for designating a siphoning destination. The details of the siphoning control table 371 will be described later.

The CPU usage rate information 372 manages information including the usage rate of each control unit in the CM. The command number information 373 manages information including the number of input/output commands being processed by each control unit in the CM.

Functional Configuration of Slave CM

The CM 3B is a device that controls output and input of data to and from the disks 4. The CM 3B includes a plurality of CAs 31, an IOC 32, a BUD 33, a main control unit 34B, a sub-control unit 35A, an inter-CM communication driver 36, and a storage unit 37B. Units of the CM 3B in the second embodiment having functions similar to those in the configuration of the CM 3A illustrated in FIG. 2 are referred to with the same reference numerals and the detailed description thereof will be omitted. The number of control units provided in the CM 3B is not limited to the example illustrated in FIG. 2. For example, the CM 3B may include a single control unit.

The main control unit 34B is a main control unit for the case where processing in the CM 3B is performed in a distributed manner. The main control unit 34B includes the siphoning processing part 51 and the information acquiring part 53. Hereinafter, description will be provided on the assumption that the main control unit 34B is a CPU. However, the main control unit 34B may be an electronic circuit such as an MPU. The main control unit 34B functions as various functional parts by, for example, executing CFW for controlling the CM 3B.

The storage unit 37B stores various data to be used for processing by the main control unit 34B or the sub-control unit 35A. For example, the storage unit 37B stores the CPU usage rate information 372 and the command number information 373.

Data Structure of Siphoning Control Table

A data structure of the siphoning control table 371 will be described with reference to FIG. 3. FIG. 3 illustrates an example of a data structure of the siphoning control table 371 used in the second embodiment. As illustrated in FIG. 3, the siphoning control table 371 stores an “active flag” item 371b, a “siphoning target CA” item 371c, a “CA dump siphoning time” item 371d, a “CPU usage rate” item 371e, and a “CA dump siphoning execution time” item 371f in association with a “CPU” item 371a. Furthermore, the siphoning control table 371 stores an “I/O command number” item 371g and an “I/O processing time” item 371h in association with the “CPU” item 371a.

The “CPU” item 371a represents a CPU corresponding to a control unit in the case where a CPU is used as the control unit. The “active flag” item 371b is a flag representing whether or not siphoning processing is being performed. For example, in the case where the siphoning processing is being performed, “ON” is set. In the case where the siphoning processing is not being performed, “OFF” is set. The “siphoning target CA” item 371c represents a siphoning target CA in the case where the siphoning processing is being performed. The “CA dump siphoning time” item 371d represents an estimated time for siphoning processing for dump data by the CPU, which is estimated on the basis of the size of the dump data of the CA. The “CPU usage rate” item 371e represents the usage rate of the CPU. The “CA dump siphoning execution time” item 371f represents an estimated time for performing siphoning processing corresponding to the usage rate of the CPU. The “I/O command number” item 371g represents the number of I/O commands being processed by the CPU. For example, the number of I/O commands for cross access and the number of I/O commands for straight access are individually set. The “I/O processing time” item 371h represents the processing time converted from the number of I/O commands for the CPU.

In the case where the “CPU” item 371a is a “main CPU of the CM#0”, for example, “OFF” is stored as the “active flag” item 371b, “−” is stored as the “siphoning target CA” item 371c, and “50” seconds (s) is stored as the “CA dump siphoning time” item 371d. Furthermore, “0.8” is stored as the “CPU usage rate” item 371e, “250” seconds (s) is stored as the “CA dump siphoning execution time” item 371f, “cross 100/straight 100” is stored as the “I/O command number” item 371g, and “2” seconds (s) is stored as the “I/O processing time” item 371h. Furthermore, in the case where the “CPU” item 371a is a “sub-CPU of the CM#7”, “CM#1-CA#0” is stored as the “siphoning target CA” item 371c. That is, in the example illustrated in FIG. 3, the sub-CPU of the CM#7 is performing processing for siphoning dump data of the CA#0 in the CM#1.

Processing Operation of Storage System

The processing operation of a storage system will now be described with reference to FIGS. 4A to 4C and FIGS. 5A to 5E. The processing operation of saving processing for dump data will be described with reference to FIGS. 4A to 4C and the processing operation of copying processing will be described with reference to FIGS. 5A to 5E.

Saving Processing for Dump Data

FIGS. 4A to 4C illustrate an example of the processing operation of saving processing for dump data. As illustrated in FIG. 4A, in the case where a degradation of the CA#1 has occurred in the CM#1, the siphoning destination designating part 52 in the CM#0, which is a master CM, selects two control units from different CMs. The case where the siphoning destination designating part 52 selects the main control unit 34B of the CM#5 and the main control unit 34B of the CM#6 is illustrated in the example of FIG. 4A.

In the main control unit 34B of the CM#1, the siphoning processing part 51 transfers identical dump data to the main control unit 34B of the CM#5 and the main control unit 34B of the CM#6. The main control unit 34B of the CM#5 and the main control unit 34B of the CM#6 respectively perform compression of the dump data and saving of the dump data into the BUD 33 in a parallel manner. Here, in the case where the usage rate of the main control unit 34B of the CM#5 exceeds a specific threshold during execution of compression of the dump data and the saving of the dump data into the BUD 33, the siphoning destination designating part 52 in the CM#0 selects a control unit of a new saving destination. Here, an example in which the siphoning destination designating part 52 of the CM#0 switches the control unit of the saving destination from the main control unit 34B of the CM#5 to the main control unit 34B of the CM#6 will be described.

Accordingly, even in the case where a degradation has occurred in a control unit of one saving CM during execution of siphoning of CA dump, a control unit of the other CM is capable of acquiring CA dump. For example, as illustrated in FIG. 4B, even in the case where a degradation has occurred in the saving CM#5 during execution of the processing, the main control unit 34B of the CM#6 acquires CA dump from the CM#1.

Furthermore, in the case where a degradation has occurred in the CM#5 during execution of siphoning processing for CA dump, the siphoning destination designating part 52 of the main control unit 34A of the CM#0, which is a master CM, newly searches for a control unit of a CM and assigns the found control unit of the CM for a saving destination. In the example illustrated in FIG. 4C, the siphoning destination designating part 52 assigns the main control unit 34B of the CM#7 for a saving destination.

Then, the siphoning processing part 51 of the main control unit 34B of the CM#1 resumes transfer of dump data to the main control unit 34B of the CM#6 and the main control unit 34B of the CM#7. Since CA dump data is sequentially transferred from a front portion, the front portion of the dump data is not transferred to the main control unit 34B of the CM#7. Thus, CA dump data not including the front portion is saved into the BUD 33 of the CM#7.

Copying Processing

FIG. 5A illustrates an example of dump data that is saved into the BUD 33 of each CM after a degradation occurs in the CM#5 serving as a saving destination of CA dump and the CM#7 is newly selected as a saving destination. FIG. 5B illustrates an example of processing for copying dump data from the BUD 33 of the CM#6 to the BUD 33 of the CM#7. FIG. 5C illustrates an example of dump data that is saved into the BUD 33 of the CM#7 after the dump data is copied from the BUD 33 of the CM#6. FIG. 5D illustrates an example of the case where after a degradation occurs in the CM#5 serving as a saving destination of CA dump, a degradation occurs also in the CM#6 serving as another saving destination of the CA dump. FIG. 5E illustrates an example of dump data that is saved into the BUD 33 of the CM#7 in the case where a degradation occurs also in the CM#6 after a degradation occurs in the CM#5.

As illustrated in FIG. 5A, dump data 100a that is collected by the time when a degradation occurs in the CM#5 is saved in the BUD 33 of the CM#5. Furthermore, dump data 100b that is collected by the time when the degradation occurs in the CM#5 and dump data 100c that is collected after the degradation occurs in the CM#5 are saved in the BUD 33 of the CM#6. Furthermore, dump data 100d that is collected after the degradation occurs in the CM#5 is saved in the BUD 33 of the CM#7. As described above, dump data not including dump data that is collected by the time when the degradation occurs in the CM#5 is saved in the BUD 33 of the CM#7.

Thus, as illustrated in FIG. 5B, in the storage system 2A, the main control unit 34B of the CM#6 transfers dump data saved in the BUD to the main control unit 34B of the CM#7 in parallel with siphoning processing for dump data. Since the front portion of the CA dump saved in the BUD 33 of the CM#6 is compressed and is small in size, transferring the dump data may be completed in a time shorter than the time for siphoning of CA dump data. In other words, the time for copying saved dump data is shorter than the time for saving CA dump. For example, the time for saving CA dump is in the order of 100 ms/MB, while the time for copying saved dump data is in the order of 1 ms/MB.

Consequently, as illustrated in FIG. 5C, the dump data 100b saved in the BUD 33 of the CM#6 and the dump data 100d collected after the degradation occurs in the CM#5 are saved in the BUD 33 of the CM#7. As described above, since CA dump data may be saved into the CM#7, even if a degradation occurs further in the CM#6 or the CM#7, CA dump may be acquired.

The case where a degradation occurs also in the CM#6 after a degradation occurs in the CM#5 is illustrated in the example of FIG. 5D. In this case, in the storage system 2A, CA dump data is saved into the CM#7. In the example illustrated in FIG. 5D, even if a degradation occurs in the CM#6 before copying to the CM#7 is performed, the storage system 2A is capable of saving a large amount of dump data as much as possible into the BUD 33 of the CM#7.

For example, as illustrated in FIG. 5E, in the storage system 2A, even after a degradation occurs in the CM#6, the dump data 100d and dump data 100e collected from the CA are saved in the BUD 33 of the CM#7.

Processing Procedure of Processing by Storage System

The processing procedure of processing by the storage system will now be described with reference to FIGS. 6A and 6B and FIGS. 7A and 7B. The processing procedure of siphoning processing for CA dump by the storage system will be described with reference to FIGS. 6A and 6B, and the processing procedure of siphoning destination designating processing by the siphoning destination designating part 52 will be described with reference to FIGS. 7A and 7B. In the description provided below, a main control unit provided in each CM is referred to as a CPU#0, and a sub-control unit provided in each CM is referred to as a CPU#1.

Siphoning Processing for CA Dump by Storage System

FIGS. 6A and 6B are sequence diagrams illustrating siphoning processing for CA dump by the storage system 2A. The case where a degradation of the CA#1 has occurred in the CM#1 and the CPU#0 of the CM#0 (referred to as CM#0-CPU#0), which is a master CM, selects the CPU#1 of the CM#5 (referred to as CM#5-CPU#1) and the CPU#0 of the CM#6 (referred to as CM#6-CPU#0) as saving destinations is illustrated in FIGS. 6A and 6B. Furthermore, the case where a degradation of the CM#5 has occurred and the CM#0-CPU#0, which is the master CM, newly selects the CPU#1 of the CM#7 (referred to as CM#7-CPU#1) as a new saving destination is illustrated in FIGS. 6A and 6B.

As illustrated in FIG. 6A, in the case where a degradation of the CA#1 has occurred in the CM#1, the CM#1-CPU#0 (referred to as CM#1-CPU#0) acquires the size of dump data of the CA#1 (S101), and requests the master CM for acquisition of a saving destination (S102).

The CM#0-CPU#0 acquires the CPU load and the number of I/O commands from each CPU in the storage system 2A (S103). Then, the CM#0-CPU#0 updates the siphoning control table (S104), selects two saving destination CPUs, and notifies the CM#1-CPU#0 of the selected saving destination CPUs (S105). Here, the description is provided on the assumption that the CM#0-CPU#0 selects the CM#5-CPU#1 and the CM#6-CPU#0 as the saving destination CPUs.

The CM#1-CPU#0 requests the CM#5-CPU#1 and the CM#6-CPU#0 to save dump data of the CA#1 (S106). Then, the CM#1-CPU#0 acquires the dump data of the CA#1 (S107), and transfers the acquired dump data of CA#1 to the CM#5-CPU#1 and the CM#6-CPU#0 (S108).

The CM#5-CPU#1 and the CM#6-CPU#0 respectively compress the dump data of the CA#1 received from the CM#1-CPU#0, and save the compressed dump data into the BUD 33 (S109, S110). The CM#5-CPU#1 and the CM#6-CPU#0 also notify the CM#1-CPU#0 of completion of the saving (S111, S112).

Here, the case where the CM#1-CPU#0 that is notified of the completion of the saving determines that all the dump data of the CA#1 has not yet been saved is illustrated in FIG. 6A. The CM#1-CPU#0 acquires the dump data of the CA#1 (S113), and transfers the acquired dump data of the CA#1 to the CM#5-CPU#1 the CM#6-CPU#0 (S114).

The CM#6-CPU#0 compresses the dump data of the CA#1 received from the CM#1-CPU#0, and saves the compressed dump data into the BUD 33 (S115). The CM#6-CPU#0 also notifies the CM#1-CPU#0 of completion of the saving (S116).

Furthermore, a degradation of the CM#5 occurs. Thus, the CM#5-CPU#1 notifies the CM#1-CPU#0 of the degradation (S117).

The CM#1-CPU#0 requests the master CM for acquisition of a siphoning destination (S118). The CM#0-CPU#0 updates the siphoning control table (S119), selects a new saving destination CPU, and notifies the CM#1-CPU#0 of the selected new saving destination CPU (S120). Here, description is provided on the assumption that the CM#0-CPU#0 selects the CM#7-CPU#1 as a new saving destination CPU.

Then, as illustrated in FIG. 6B, the CM#1-CPU#0 requests the CM#7-CPU#1 to save the dump data of the CA#1 (S121). Then, the CM#1-CPU#0 requests the CM#6-CPU#0 to copy the dump data of the CA#1 (S122). That is, the CM#1-CPU#0 causes the CM#6-CPU#0 to transfer the dump data of the CA#1 saved in the CM#6-CPU#0 to the CM#7-CPU#1, and causes the CM#7-CPU#1 to save the transferred dump data.

The CM#6-CPU#0 transfers copy data to the CM#7-CPU#1 (S123). Then, the CM#7-CPU#1 saves the received copy data into the BUD 33 (S124). The CM#6-CPU#0 determines whether or not copying has been completed (S125). In the case where the CM#6-CPU#0 determines that copying has been completed (YES in S125), the CM#6-CPU#0 notifies the CM#7-CPU#1 of completion of the copying (S126). In the case where the CM#6-CPU#0 determines that copying has not yet been completed (NO in S125), the CM#6-CPU#0 proceeds to S124.

The CM#1-CPU#0 acquires dump data of the CA#1 (S127), and transfers the acquired dump data of the CA#1 to the CM#6-CPU#0 and the CM#7-CPU#1 (S128).

The CM#6-CPU#0 and the CM#7-CPU#1 respectively compress the dump data of the CA#1 received from the CM#1-CPU#0, and save the compressed dump data into the BUD 33 (S129, S130). The CM#6-CPU#0 and the CM#7-CPU#1 also notify the CM#1-CPU#0 of completion of the saving (S132).

The CM#1-CPU#0 that is notified of the completion of the saving determines whether to terminate the saving processing (S133). In other words, the CM#1-CPU#0 determines whether or not all the dump data of the CA#1 has been saved. In the case where the CM#1-CPU#0 determines to terminate saving processing (YES in S133), the CM#1-CPU#0 notifies the CM#0-CPU#0 of completion of the siphoning processing (S134).

The CM#0-CPU#0 that is notified of the completion of the siphoning processing updates the siphoning control table (S135). In the case where the CM#1-CPU#0 determines not to terminate the saving processing (NO in S133), the CM#1-CPU#0 proceeds to S127.

Siphoning Destination Designating Processing

FIGS. 7A and 7B are flowcharts illustrating the processing procedure of siphoning destination designating processing in the second embodiment. Here, it is assumed that an index is assigned in advance for each CPU that may serve as a siphoning destination. For example, “0” is assigned for the main CPU of the CM#0, “1” is assigned for the sub-CPU of the CM#0, “2” is assigned for the main CPU of the CM#1, and “3” is assigned for the sub-CPU of the CM#1.

For example, upon receiving a command for acquiring a siphoning destination CPU including the size of dump data of a CA in which an error has occurred, the siphoning destination designating part 52 calculates a standard siphoning time (CA dump siphoning time) on the basis of the size of the dump data (S201). The CA dump siphoning time is calculated using Equation (1). Then, the siphoning destination designating part 52 stores the calculated CA dump siphoning time into the siphoning control table 371.

Then, the siphoning destination designating part 52 acquires the usage rate of CPU and the number of I/O commands from each CPU (S202). The usage rate of each CPU is stored in the CPU usage rate information 372 for each CM. The number of I/O commands for each CPU is stored in the command number information 373 for each CM.

Then, the siphoning destination designating part 52 calculates a standard siphoning execution time (CA dump siphoning execution time) of each CPU on the basis of the standard siphoning time and the usage rate of each CPU (S203). The CA dump siphoning execution time is calculated using Equation (2). Then, the siphoning destination designating part 52 stores the calculated CA dump siphoning execution time for each CPU into the siphoning control table 371.

Furthermore, the siphoning destination designating part 52 calculates the I/O processing time for each CPU on the basis of the number of I/O commands (S204). Then, the siphoning destination designating part 52 stores the calculated I/O processing time for each CPU into the siphoning control table 371.

Then, the siphoning destination designating part 52 sets CANDIDATE-CPU to an initial value (for example, 0xFF) (S205). The CANDIDATE-CPU is a variable representing a CPU candidate for a siphoning destination CPU, and the value of the index assigned for the CANDIDATE-CPU is set. Furthermore, the siphoning destination designating part 52 sets SHORTEST-SAVING-TIME-CPU to an initial value (for example, 0xFF) (S206). The SHORTEST-SAVING-TIME-CPU is a variable representing a CPU whose dump data saving time is the shortest, and the value of the index assigned for the CPU whose saving time is the shortest is set.

Then, the siphoning destination designating part 52 sets INDEX to the value 0, which is the index of the CPU serving as a siphoning destination, and sets CHECK-CPU to the value set for the INDEX (S207). The INDEX is a variable, and the value of the index assigned for a CPU is set. The CHECK-CPU is variable representing a CPU that is checked whether to be a siphoning destination, and the value of the index assigned for the CPU is set.

Then, the siphoning destination designating part 52 determines whether or not the value set for the INDEX is equal to the maximum number of the CPUs that may serve as a siphoning destination (S208).

In the case where it is determined that the value set for the INDEX is not equal to the maximum number of the CPUs (NO in S208), the siphoning destination designating part 52 proceeds to S214.

The siphoning destination designating part 52 determines whether or not the active flag of the CHECK-CPU is ON, on the basis of the active flag stored in the siphoning control table 371 (S214). In the case where it is determined that the active flag of the CHECK-CPU is ON (YES in S214), the siphoning destination designating part 52 proceeds to S222 to check the next CPU.

In the case where it is determined that the active flag of the CHECK-CPU is not ON (NO in S214), the siphoning destination designating part 52 determines whether or not the SHORTEST-SAVING-TIME-CPU is set to the initial value (S215). In the case where it is determined that the SHORTEST-SAVING-TIME-CPU is set to the initial value (YES in S215), the siphoning destination designating part 52 proceeds to S217 to set the CHECK-CPU for the SHORTEST-SAVING-TIME-CPU.

In the case where it is determined that the SHORTEST-SAVING-TIME-CPU is not set to the initial value (NO in S215), the siphoning destination designating part 52 determines whether or not the CA siphoning execution time of the CHECK-CPU is shorter than or equal to the CA siphoning execution time of the SHORTEST-SAVING-TIME-CPU (S216). In the case where it is determined that the CA siphoning execution time of the CHECK-CPU is shorter than or equal to the CA siphoning execution time of the SHORTEST-SAVING-TIME-CPU (YES in S216), the siphoning destination designating part 52 proceeds to S217 to set the CHECK-CPU for the SHORTEST-SAVING-TIME-CPU.

The siphoning destination designating part 52 sets the CHECK-CPU for the SHORTEST-SAVING-TIME-CPU (S217). That is, the siphoning destination designating part 52 sets the CHECK-CPU currently being checked for as the CPU whose saving time is the shortest among the checked CPUs.

In the case where the CA siphoning execution time of the CHECK-CPU is longer than the CA siphoning execution time of the SHORTEST-SAVING-TIME-CPU (NO in S216), the siphoning destination designating part 52 determines whether or not the CA siphoning execution time of the CHECK-CPU is shorter than or equal to a specified siphoning completion time (S218). The specified siphoning completion time represents a specified time allowed for siphoning. In the case where it is determined that the CA siphoning execution time of the CHECK-CPU is longer than the specified siphoning completion time (NO in S218), the siphoning destination designating part 52 proceeds to S222 to check the next CPU.

In the case where it is determined that the CA siphoning execution time of the CHECK-CPU is shorter than or equal to the specified siphoning completion time (YES in S218), the siphoning destination designating part 52 determines whether or not the CANDIDATE-CPU is set to the initial value (S219). In the case where it is determined that the CANDIDATE-CPU is not set to the initial value (NO in S219), the siphoning destination designating part 52 determines whether or not the I/O processing time of the CHECK-CPU is shorter than or equal to the I/O processing time of the CANDIDATE-CPU (S220). In the case where it is determined that the I/O processing time of the CHECK-CPU is longer than the I/O processing time of the CANDIDATE-CPU (NO in S220), the siphoning destination designating part 52 proceeds to S222 to check the next CPU.

In the case where the CANDIDATE-CPU is set to the initial value (YES in S219) or the I/O processing time of the CHECK-CPU is shorter than or equal to the I/O processing time of the CANDIDATE-CPU (YES in S220), the siphoning destination designating part 52 proceeds to S221. The siphoning destination designating part 52 sets the CHECK-CPU for the CANDIDATE-CPU (S221). That is, the siphoning destination designating part 52 sets the CHECK-CPU currently being checked for as the CPU whose CA siphoning execution time is shorter than the specified siphoning completion time and whose I/O processing time is the shortest among the checked CPUs.

The siphoning destination designating part 52 increments the INDEX by 1, and updates the CHECK-CPU to the Index (S222). Then, the siphoning destination designating part 52 proceeds to S208.

In the case where it is determined in S408 that the value set for the INDEX is equal to the maximum number of CPUs that may serve as a siphoning destination (YES in S208), the siphoning destination designating part 52 determines whether or not the CANDIDATE-CPU is set to the initial value (S209). In the case where it is determined that the CANDIDATE-CPU is not set to the initial value (NO in S209), the siphoning destination designating part 52 designates the CANDIDATE-CPU as a siphoning destination (S210), and terminates the siphoning destination designating processing.

In the case where it is determined that the CANDIDATE-CPU is set to the initial value (YES in S209), the siphoning destination designating part 52 determines whether or not the SHORTEST-SAVING-TIME-CPU is set to the initial value (S211). If there is no CPU whose CA siphoning execution time is shorter than the specified siphoning completion time, the SHORTEST-SAVING-TIME-CPU is still set to the initial value. In the case where it is determined that the SHORTEST-SAVING-TIME-CPU is not set to the initial value (NO in S211), the siphoning destination designating part 52 designates the SHORTEST-SAVING-TIME-CPU as a siphoning destination (S212), and terminates the siphoning destination designating processing.

In the case where it is determined that the SHORTEST-SAVING-TIME-CPU is set to the initial value (YES in S211), the siphoning destination designating part 52 determines that there is no candidate CPU for a siphoning destination (S213), and terminates the siphoning destination designating processing.

Effects of Second Embodiment

As described above, the storage system 2A according to the second embodiment selects a CPU that is processing the smallest number of I/O commands, on the basis of the number of I/O commands being processed by each CPU, to perform siphoning processing for CA dump. That is, the storage system 2A according to the second embodiment separates a collecting CM and a saving CM from each other. Accordingly, the storage system 2A according to the second embodiment is capable of reducing the influence of processing for siphoning dump data of a CA on the performance of the storage system 2A.

Furthermore, since the storage system 2A according to the second embodiment separates a collecting CM and a saving CM from each other, the number of device components regarding siphoning processing for CA dump increases. By duplicating a control device that performs siphoning of CA dump, even in the case where an error has occurred in one control device during execution of the siphoning processing for CA dump, the storage system 2A according to the second embodiment is capable of recording the dump data with certainty. In other words, although separation between a collecting CM and a saving CM increases the number of device components regarding siphoning processing for CA dump, the storage system 2A according to the second embodiment is capable of recording dump data with certainty. Here, a single path is used for communication between a CA and a CM including the CA, and four paths are used for communication between CMs. Thus, even for control performed in the case where a CPU of a collecting CM and a CPU of a saving CM are separated from each other, CPUs of two CMs may be designated as CPUs of saving CMs.

Furthermore, in the case where a degradation of a control device performing siphoning has occurred, the storage system 2A according to the second embodiment selects a new saving destination and causes the selected new saving destination to perform siphoning. Accordingly, even in the case where a degradation of a saving CM has occurred during execution of siphoning processing for CA dump, the storage system 2A according to the second embodiment is capable of acquiring CA dump data. Furthermore, even in the case where degradations have occurred in a plurality of saving CMs, the storage system 2A according to the second embodiment is capable of saving a larger amount of dump data.

In the storage system 2A according to the second embodiment, the siphoning destination designating part 52 may be provided in the CM 3B. In this case, the storage unit 37B of the CM 3B stores the siphoning control table 371.

Other Embodiments

Various other embodiments may be implemented. Other embodiments will be described.

System Configuration

All or part of the processing that has been described as being automatically performed in the foregoing embodiments may be performed manually. Alternatively, all or part of the processing that has been described as being manually performed may be performed automatically in a known method. Furthermore, the processing procedures, control procedures, and specific names illustrated in the description and drawings may be changed unless otherwise specified.

Furthermore, in the storage system 2 according to the first embodiment, in the case where an error has occurred in a CA 31, the main control unit 34 may select a saving destination for dump data collected from the CA 31 on the basis of the usage rates and the number of I/O commands of control units in the storage system 2.

Furthermore, the order of the processing procedure of the individual processing operations described in the foregoing embodiments may be changed in accordance with various loads and use conditions. Furthermore, component parts in the drawings are illustrated in terms of functional concepts and may not be physically configured as illustrated. Furthermore, all or part of the processing functions in the individual devices may be implemented by a CPU analyzing and executing a program or may be implemented as hardware by wired logic.

All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.

STORAGE SYSTEM AND METHOD FOR CONTROLLING STORAGE SYSTEM

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

US Classifications

International Classifications

Abstract

Description

Claims

Priority Claims (1)