1. Field of the Invention
The present invention relates to a technology for adding/changing a backup server in a plurality of server groups configured with a plurality of servers, with a capability of recovering a server group at low cost in an expedited manner.
2. Description of the Related Art
An information processing system that performs various tasks is configured with a plurality of servers, in which a backup server is prepared and to be added to the information processing system so that when a failure occurs in a server included in the information processing system or a load on the server increases, a failed server is replaced with the backup server or loads on the servers are distributed to the backup server.
However, upon adding a new server, it is necessary to install software for performing various processings or to perform various environmental settings to the server to be added. Therefore, there is a problem that it takes time before a new server is added.
Furthermore, because an installation of the software or various environmental settings are manually performed, if there is a mistake in operation, a problem occurs that the installation of the software or the various environmental settings are not performed properly and the server added does not work.
To cope with the above problems, Japanese Patent Application Laid-open No. H1-85555 discloses a technology for replacing an operating server with a backup server in an expedited manner when abnormality occurs in the operating server, by dynamically copying a task class of the operating server to a task class of the backup server to continuously keep the task classes of the servers uniform.
With the conventional technology, however, it is necessary to continuously operate the backup server and to keep a state of the backup server same as that of the operating server even when the operating server is not failed. As a result, there is a problem that extra costs are required.
Therefore, there is a need that the backup server does not continuously operate and can be added or used as a replacement in an expedited manner only when a failure occurs in the operating server or a load on the operating server increases.
It is an object of the present invention to at least partially solve the problems in the conventional technology.
A computer-readable recording medium according to one aspect of the present invention stores therein a computer program for adding/changing a backup server in a plurality of server groups configured with a plurality of servers. The computer program causes a computer to execute setting including acquiring setting information, which is information on a setting of a software, a network, and a storage, for a server group, and performing a setting corresponding to the server group on the backup server in advance, based on the setting information; and adding, when an abnormality occurs in the server group or when a load on the server group increases, the backup server for which the setting is performed at the setting to the server group.
A method according to another aspect of the present invention is for adding/changing a backup server in a plurality of server groups configured with a plurality of servers. The method includes setting including acquiring setting information, which is information on a setting of a software, a network, and a storage, for a server group, and performing a setting corresponding to the server group on the backup server in advance, based on the setting information; and adding, when an abnormality occurs in the server group or when a load on the server group increases, the backup server for which the setting is performed at the setting to the server group.
An apparatus according to still another aspect of the present invention is for adding/changing a backup server in a plurality of server groups configured with a plurality of servers. The apparatus includes a setting processing unit that acquires setting information, which is information on a setting of a software, a network, and a storage, for a server group, and performs a setting corresponding to the server group on the backup server in advance, based on the setting information; and an adding processing unit that adds, when an abnormality occurs in the server group or when a load on the server group increases, the backup server for which the setting is performed by the setting processing unit to the server group.
The above and other objects, features, advantages and technical and industrial significance of this invention will be better understood by reading the following detailed description of presently preferred embodiments of the invention, when considered in connection with the accompanying drawings.
Exemplary embodiments of the present invention are explained in detail below with reference to the accompanying drawings. The present invention is not limited to the embodiments.
The web servers 41 to 49 are servers that provide contents to be browsed by web browsers to client terminals via the Internet. The AP servers 51 to 56 are servers that take over execution of information processes requested by the web servers 41 to 49 that have received an information processing request from a user.
The DB servers 61 to 63 are servers that manage accesses to database upon receiving requests for accessing the database from the AP servers 51 to 56. The storages 71 to 79 are storage devices to be connected via a SAN (Storage Area Network) to the web servers 41 to 49, the AP servers 51 to 56, and the DB servers 61 to 63.
With operation management according to the present invention, a resource group that contains servers or storages having a uniform physical wire connection to other devices is managed as a domain in a LAN (local area network) or a SAN.
For example, in the case shown in
In this case, the web servers 41 to 49 that belong to the web domain 4 have uniform connections to other devices, the AP servers 51 to 56 that belong to the AP domain 5 have uniform connections to other devices, the DB servers 61 to 63 that belong to the DB domain 6 have uniform connections to other devices, and the storages 71 to 79 that belong to the storage domain 7 have uniform connections to other devices.
With the operation management, unused ones of the web servers 41 to 49, the AP servers 51 to 56, the DB servers 61 to 63, and the storages 71 to 79 are registered to a pool 3 for each domain. The web servers 41 to 49, the AP servers 51 to 56, the DB servers 61 to 63, and the storages 71 to 79 are assigned to each of the tasks 1 and 2 as appropriate.
For example, in the example of
If load on the web servers 42, 43, 49, the AP servers 51, 52, 53, and the DB servers 61, 62 assigned to the tasks 1 and 2 increases, or if storage capacity of the storages 77 to 79 is not enough, the web servers 41, 44 to 48, the AP servers 54 to 56, the DB server 63, and the storages 71 to 76 registered to the pool 3 are added as servers available for the task.
Specifically, by introducing a software required for the web servers 41, 44 to 48, the AP servers 54 to 56, and the DB server 63 registered to the pool 3 and automatically executing setting of the network and the like, the web servers 41, 44 to 48, the AP servers 54 to 56, and the DB server 63 are added as the servers available for the task.
To add the storages 71 to 76, by automatically executing setting of logical volumes and setting of the network to the storages 71 to 76, the storages 71 to 76 are added as the storages 71 to 76 available for the task.
For example, in
According to the embodiment, as shown in
Moreover, according to the resource allocation management method, if the web servers 42, 43, 49, the AP servers 51 to 53, the DB servers 61, 62, or the storages 77 to 79 used for the tasks 1 and 2 are not used for a long time, the web servers 42, 43, 49, the AP servers 51 to 53, the DB servers 61, 62, and the storages 77 to 79 are excluded from the servers available for the task and registered to the pool 3.
The web servers 42, 43, 49, the AP servers 51 to 53, the DB servers 61, 62, and the storages 77 to 79 registered to the pool 3 are to be used if the load on the web servers 42, 43, 49, the AP servers 51 to 53, and the DB servers 61, 62, used for other tasks 1 and 2 increases or if the storage capacity of the storages 77 to 79 is not enough.
Specifically, by deleting the software introduced to the web servers 42, 43, 49, the AP servers 51 to 53, and the DB servers 61, 62 and automatically changing setting of the network and the like, the web servers 42, 43, 49, the AP servers 51 to 53, and the DB servers 61, 62 are excluded from the servers available for the tasks 1 and 2 and registered to the pool 3.
Moreover, to exclude the storages 77 to 79 from the storages available for the tasks 1 and 2 and to register them to the pool 3, by automatically executing setting of the network to the storages 77 to 79, the storages 77 to 79 can be excluded from the storages available for the tasks 1 and 2 and registered to the pool 3.
For example, in
As shown in
Furthermore, the site management server 20 is connected over the network to a router 80 that belongs to an edge domain 180 via the FW 40. The site management server 20 is also connected over the network to storages 160a to 160c that belong to a storage domain 220, and to a storage 160d that is pooled via the FW 40.
The domain management server 50 is connected over the network to an SLB (server load balancer) 100 and to servers 110a to 110c that belong to a web domain 190.
Furthermore, the domain management server 60 is connected over the network to an FW 120, an SLB 130, servers 140a to 140c that belong to an AP domain 200, servers 150a to 150c that belong to a DB domain 210.
The storages 160a to 160c that belong to the storage domain 220, and the storage 160d that is pooled are also connected via a SAN 170 to the servers 110a to 110c that belong to the web domain 190, the servers 140a to 140c that belong to the AP domain 200, and the servers 150a to 150c that belong to the DB domain 210.
In this case, the operation management client 10 is a client device that receives various settings related to the resource-change processing from the user, sends the setting information to, the site management server 20, receives various output results from the site management server 20, and displays the results on a monitor or the like.
The site management server 20 is a server that performs the resource-change processing as explained in connection with
The system resource manager 21 is a managing unit that receives various setting information related to the resource-change processing from the operation management client 10, and operates resources in cooperation with the server RM 22, the software RM 23, the network RM 24, and the storage RM 25. In addition, the system resource manager 21 performs data reception and data transmission between the domain management servers 50 and 60.
The server RM 22 is a managing unit that performs a boot and a shutdown of each of the servers 110a to 110c, 140a to 140c, and 150a to 150c, a collection of information about hardware, a setting, and the like. The server RM 22 performs the above processes in cooperation with a server sub RM 52 of the domain management server 50, and a server RM agent 112a of the server 110a.
The software RM 23 is a managing unit that performs software installation, setting, collection of information about the software, and the like for each of the servers 110a to 110c, 140a to 140c, and 150a to 150c. The software RM 23 performs the above processes in cooperation with a software sub RM (resources manager) 53 of the domain management server 50, and a software RM agent 113a of the server 110a.
The network RM 24 is a managing unit that performs information collection, setting, and the like related to the network. The network RM 24 performs the above processes in cooperation with a network sub RM 54 of the domain management server 50, and a network RM agent 114a of the server 110a.
The storage RM 25 is a managing unit that performs information collection, setting, and the like related to the storages 160a to 160c that belong to the storage domain 220, and relate to the storage 160d that is pooled. The storage RM 25 manages the storages 160a to 160c and the storage 160d pooled without involving the domain management servers 50 and 60.
The system resource DB 26 is a database that contains various resource information managed by the system resource manager 21, the server RM 22, the software RM 23, the network RM 24, and the storage RM 25. Details of stored data are explained later.
The AP-management control unit 27 is a processing unit that controls and manages an AP managing unit 116a. More specifically, the AP-management control unit 27 sends a request for executing process related to an application such as installation and setting to the AP managing unit 116a. Functions of the AP-management control unit 27 are realized by executing middleware installed on the site management server 20.
The domain management servers 50 and 60 are servers that manage resources in a domain or a plurality of domains. The domain management server 50 includes a system resource domain manager 51, the server sub RM 52, the software sub RM 53, the network sub RM 54, and a domain resource DB 55.
The domain management server 60 includes the same function units as the function units of the domain management server 50, and therefore, the function units of the domain management server 60 are not shown in
The system resource domain manager 51 is a managing unit that performs information collection, setting process, and the like related to resources that belong to each of the domains in cooperation with the server sub RM 52, the software sub RM 53, and the network sub RM 54.
Furthermore, the system resource domain manager 51 performs data reception and data transmission to and from networking equipment such as the site management server 20, an FW 90, and the SLB 100, as well as to and from the servers 110a to 110c to be managed.
The server sub RM 52 is a managing unit that performs boot, shutdown, collection of information about hardware, setting, and the like in cooperation with the server RM 22 and the server RM agent 112a.
The software sub RM 53 is a managing unit that performs software installation, setting, collection of information about software, and the like for each of the servers 110a to 110c in cooperation with the software RM 23 and the software RM agent 113a.
The network sub RM 54 is a managing unit that performs information collection, setting, and the like related to a network in cooperation with the network RM 24 and a network RM agent 114a.
The domain resource DB 55 is a database that stores therein information acquired from the servers 110a to 110c and the system resource DB 26, when the server sub RM 52, the software sub RM 53, or the network sub RM 54 collects various information or specifies settings related to the servers 110a to 110c to be managed. In addition, the domain resource DB 55 stores therein a virtual OS (operating system) used for network boot of the servers 110a to 110c.
The router 80 is networking equipment that performs routing of data packets in data communication via the Internet 70. The FWs 30, 40, 90, and 120 are networking equipments that prevent unauthorized access to each of the servers 110a to 110c, 140a to 140c, and 150a to 150c.
The SLBs 100 and 130 are load balancers that distribute and transfer information-processing requests for the servers 110a to 110c or 140a to 140c to a plurality of the servers 110a to 110c or 140a to 140c. Although switches are also connected in upstream sides and downstream sides of the SLBs 100 and 130, the switches are not shown in
The servers 110a to 110c, 140a to 140c, and 150a to 150c are servers that perform various information processes. The server 110a includes a resource manager agent 111a, the server RM agent 112a, the software RM agent 113a, the network RM agent 114a, a storage RM agent 115a, and the AP managing unit 116a.
The servers 110b, 140a, 140b, 150a, and 150b include the same function units as those of the server 110a. Therefore, the function units of the servers 110b, 140a, 140b, 150a, and 150b are not shown in
The servers 110c, 140c, and 150c are pooled servers and do not include the above function units. When the servers 110c, 140c, and 150c are set as servers available for the task, the function units are implemented by installing computer programs for implementing the function units into the servers 110c, 140c, and 150c and executing the computer programs.
The resource manager agent 111a is an agent that receives a request for executing process such as setting and information collection from the domain management server 50 of the system resource domain manager 51 for the server 110a, and performs processes in cooperation with the server RM agent 112a, the software RM agent 113a, the network RM agent 114a, and the storage RM agent 115a.
The server RM agent 112a is an agent that performs a boot and a shutdown of the server 110a, a collection of information about hardware, a setting, and the like. The software RM agent 113a is an agent that performs software installation, setting, and collection of information about software for the server 110a.
The network RM agent 114a is an agent that performs information collection, setting, and the like related to a network connected to the server 110a. The storage RM agent 115a is an agent that performs information collection, setting, and the like related to a storage connected to the server 110a.
The storages 160a to 160c are storages that are used by the servers 110a to 110c that belong to the web domain 190, the servers 140a to 140c that belong to the AP domain 200, and the servers 150a to 150c that belong to the DB domain 210, and are configured with RAID devices. The storage 160d is a pooled storage.
It is allowable to set a VLAN (virtual local area network) as a network that connects between the servers 110a to 110c that belong to the web domain 190, the servers 140a to 140c that belong to the AP domain 200, and the servers 150a to 150a that belong to the DB domain 210.
A processing procedure of the resource-change processing according to the embodiment is explained below.
In this case, programs that causes a computer to execute functions of the system resource manager 21, the server RM 22, the software RM 23, the network RM 24, the storage RM 25, and the AP-management control unit 27 are installed in advance to the site management server 20, programs that causes a computer to execute functions of the system resource domain manager 51, the server sub RM 52, the software sub RM 53, and the network sub RM 54 are installed in advance to the domain management servers 50 and 60, and programs that causes a computer to execute functions of the resource manager agent 111a, the server RM agent 112a, the software RM agent 113a, the network RM agent 114a, the storage RM agent 115a, and the AP managing unit 116a are installed in advance to each of the servers 110a, 110b, 140a, 140b, and 150a, 150b.
As shown in
The site is information that identifies a site that includes a resource to be managed. The site management server is information on the site management server 20 set to manage the site. The domain management server is information on the domain management servers 50 and 60 set to manage domains set in the site.
The domain management server is information equivalent to the domain management server explained in connection with
The management subnet is the same information as the management subnet explained in connection with
At step S101, the system resource manager 21 receives information on site, site management server, and domain management server, which are set by the user by operating the operation management client 10, and registers received information on the site data 300 shown in
The system resource manager 21 receives information on domain management server and management subnet, which are set by the user by operating the operation management client 10, and registers received information on the domain management server data 310 shown in
Thereafter, the system resource manager 21 registers information on network address, netmask, and default gateway, which correspond to the management subnet explained in connection with
In addition, the system resource manager 21 notifies the AP-management control unit 27 of occurrence of an event such as addition to or deletion from the servers 110a to 110c, 140a to 140c, and 150a to 150c, and sets commands for executing various processes in cooperation with the AP-management control unit 27.
The middleware name is information on middleware with which the system resource manager 21 performs processes. The target event is information on events that the system resource manager 21 requests the middleware to execute. The timing is information on timing at which the system resource manager 21 transmits a request for executing processes to the middleware (before or after a process for the target event).
The location is information on locations where the middleware executes a command (a “manager” or an “agent”). The “manager” indicates that the command is executed on the site management server 20, while the “agent” indicates that the command is executed on the servers 110a to 110c, 140a to 140c, and 150a to 150c to be managed. The execution command is information on commands that notifies the middleware of occurrence of various events.
Referring back to
The server domain data 340 contains information on server domain, server architecture, and management subnet. The server domain is information on a domain to which the servers 110a to 110c, 140a to 140c, and 150a to 150c belong.
The server architecture is information on a CPU (central processing unit) architecture of the servers 110a to 110c, 140a to 140c, and 150a to 150c that belong to each of the server domains. The management subnet is information equivalent to the management subnet shown in
At step S102, the system resource manager 21 receives information on settings of the server domains and the server architectures specified by the user by operating the operation management client 10, and registers received information on the server domain data 340. The server domains are set in units of the management subnet set at step S101.
Furthermore, at step S102, the system resource manager 21 sets server groups that belong to each of the server domains, and sets pool groups shared between the server groups and pool groups exclusive to specific server groups.
In this case, the server groups are created by classifying servers in the same server domain into one or more groups. The pool group is a pool of the servers assigned to each of the server groups.
The pool group is information that identifies a pool of each of the above described servers. The type is information that indicates whether the pool group is to be shared by a plurality of the server groups or to be exclusively permitted for usage by specific server groups. The server domain is the same information as the server domain name explained in connection with
The system resource manager 21 assigns the pool group to each of the server domains. When the server domain includes a plurality of the sever groups, the system resource manager 21 assigns the pool group exclusive to the server groups.
Thereafter, the system resource manager 21 receives information on storage domains set by the user by operating the operation management client 10, and registers received information on the system resource DB 26 as storage domain data 360 explained below.
Furthermore, the system resource manager 21 receives information on a network sub domain set by the user by operating the operation management client 10, and registers the information on the system resource DB 26 as network sub domain data 470 described below.
In this case, the network sub domain is a sub domain obtained by dividing a network domain to which a plurality of network devices that connect servers that belong to different server domains belong.
In this case, the switches 430a and 430b configure a Web/back sub domain 420, the switches 450a and 450b configure an AP/front sub domain 440. The Web/back sub domain 420, the AP/front sub domain 440, the SLB 460a, and the SLB 460b configure a Web/AP network domain 410.
The network sub domain is information that identifies the network sub domain explained in connection with
The system resource manager 21 receives information on the network domain set by the user by operating the operation management client 10, and registers the information on the system resource DB 26 as network domain data 480 described below.
The network domain is information that identifies the network domain explained in connection with
The connection system is information on a system for connecting the network devices, such as the switches 430a and 430b that belong to the front sub domain, to the network devices, such as the switches 450a and 450b that belong to the back sub domain. For example, the above systems can include a system requiring a load balancer for a connection and a system requiring a firewall for a connection.
The back sub domain, as shown in
Moreover, the system resource manager 21 receives information on a connection device for the network sub domain set by the user by operating the operation management client 10, registers the information on the system resource DB 26 as load distributing apparatus data 490 described below. The connection device for the network sub domain indicates devices such as the SLBs 460a and 460b described in
The load distributing apparatus name is a name that identifies a connection device for the network sub domain. The management IP is information on an IP address assigned to each of the connection devices for administrating the connection device. The model is information on a model of the connection device.
The SNMP (Simple Network Management Protocol) community is information that identifies an SNMP community to which the domain management servers 50 and 60 and the site management server 20 that manage the connection devices, and an SNMP community to which the connection devices belong. The ID/password is information on an ID and a password required for accessing the connection devices.
The system resource manager 21 receives information on the network sub group set by the user by operating the operation management client 10, registers the information on the system resource DB 26 as network sub group data 660 described below.
In this case, the network sub group is a plurality of networks obtained by dividing the networks connecting between server groups that belong to different server domains.
In this case, the servers 520a and 520b configure a A_Web server Group 530, the servers 520c and 520d configure a B_Web server Group 540, the servers 560a and 560b configure a server group A_AP 570, and the servers 560c and 560d configure a server group B_AP 580.
A network that connects the A_Web server Group 530 to the SLB 600a configures a network sub group A_Web_Back 620, a network that connects the B_Web server Group 540 to the SLB 600b configures a network sub group B_Web_Back 630, a network that connects the SLB 600a to the server group A_AP 570 configures a network sub group A_AP_Front 640, and a network that connects the SLB 600b to the server group B_AP 580 configures a network sub group B_AP_Front 650.
The network sub group is a name that identifies the network sub groups described in the example in connection with
The subnet is information on a network address and a subnet mask assigned to the network sub group. The subnet for redundancy is information on the network address and the subnet mask assigned to the network constituted of a redundant data communication line added as an extra, when the network that belongs to the network sub group is made redundant using a plurality of data communication lines.
After that, the system resource manager 21 receives information on association between server domains set by the user by operating the operation management client 10, registers the information on the system resource DB 26 as inter-server-domain link data 670 described below.
The front server domain is information that indicates a server domain closer to the Internet 70 among the server domains across the network domain shown in
Moreover, the system resource manager 21 receives information on association between server domain and storage domain set by the user by operating the operation management client 10, and registers the information on the system resource DB 26 as inter-server/storage-domain link data 680 described below
Referring back to
When the user selects a management subnet on which the server is registered by operating the operation management client 10, the system resource manager 21 receives information on the management subnet selected by the user.
The system resource manager 21 also receives information on servers to be managed, which is input by the user by operating the operation management client 10, from the operation management client 10, and stores received information in the domain resource DB 55 of the domain management server 50 as network boot server data 690 explained below. Subsequently, the servers registered are network booted, and registered as the server resources after various information on the severs are acquired.
The MAC address is information on a MAC address of the server. The IP address is information on an IP addresses assigned to the server. The host name is information on a host name assigned to the server.
In this case, upon receiving information on MAC address of the network boot server input by the user, the system resource manager 21 automatically assigns the IP address and the host name to the server corresponding to the MAC address.
The system resource manager 21 performs network boot on the server to which the IP address and the host name are assigned, by using the virtual OS stored in the domain resource DB 55 of the domain management server 50, in cooperation with the system resource domain manager 51 of the domain management server 50.
The server sub RN 52, the resource manager agent 111a, and the server RM agent 112a work together to collect information on hardware of the server and transmit collected information to the system resource domain manager 51.
Thereafter, the system resource manager 21 acquires information on hardware of the server from the system resource domain manager 51, and stores acquired information in the system resource DB 26 as management target server data 700 explained below.
When the user inputs, by operating the operation management client 10, setting information indicating whether SAN boot is to be performed, in which a server is booted by the storages 160a to 160d connected via the SAN 170, the system resource manager 21 receives the setting information and registers the setting information on the management target server data 700.
The server name is a name that identifies a server to be managed. The IP address is an IP address that is assigned to the server. The MAC address is a MAC address of the server. The server architecture is information on CPU architecture of the server. The model is information that indicates the model of the server. The SAN boot is setting information that indicates whether the SAN boot is performed. The SAN boot is a booting of the server from the storages 160a to 160d connected to the server via the SAN 170. The status is information that indicates whether an abnormality occurs in the server.
Although it is assumed herein that the user specifies the MAC address of a network boot server, the server can be automatically selected. Specifically, when the user sets information that specifies the number of servers to be automatically selected by operating the operation management client 10, the system resource manager 21 receives the information from the operation management client 10.
The system resource manager 21 selects servers of specified number, and registers information on an IP address and a host name of the servers on the network boot server data 690 shown in
In cooperation with the system resource domain manager 51 in the domain management server 50, the system resource manager 21 performs network boot on the servers assigned the IP address and the host name using the virtual OS stored in the domain resource DB 55 in the domain management server 50.
With the cooperation of the server sub RM 52, the resource manager agent 111a, and the server RM agent 112a, information on the MAC address, server architecture, model, and status of each server is collected and transmitted to the system resource domain manager 51.
After that, the system resource manager 21 obtains the information on the MAC address, server architecture, model, and status of each server from the system resource domain manager 51. The system resource manager 21 stores the information in the system resource DB 26 as the management target server data 700.
Subsequently, the system resource manager 21 registers a storage device to be managed. Examples of the storage device include FC switch and RAID device.
Specifically, when the user inputs information on the IP address of a storage to be registered as a management target with respect to each management subnet shown in
After that, the system resource manager 21 adds the servers registered on the management target server data 700 shown in
Referring to the management target server data 700 shown in
The system resource manager 21 retrieves the management target server data 700 shown in
Moreover, the system resource manager 21 checks a wire connection status of the network of the server that is added to the server domain. Specifically, the system resource manager 21 reads the inter-server-domain link data 670 shown in
The system resource manager 21 reads the network domain data 480 shown in
After that, the system resource manager 21 reads the network sub domain data 470 shown in
The system resource manager 21 requests the network RM 24 and the network sub RM 54 to check wire connections between servers and switches. In addition, the network RM 24 and the network sub RM 54 request the network RM agent 114a to check the wire connections between servers and switches, and acquire a check result.
When no error occurs in the wire connections between servers and switches, the system resource manager 21 associates information on the server with the pool group explained in connection with
The server name is the same information as described in connection with
The storage sub-group name is information that identifies a storage group when storages on the same storage domain are classified into one or more groups and assigned to each server in the server group. At this point, information on the storage sub-group has not been registered. The accessibility is information that indicates whether a server is allowed to access storages. At this point, information on the accessibility has not been registered.
After registering the saver name and the pool group name on the provisioning configuration data 710, the system resource manager 21 registers the storage device, which has been previously registered, on a storage domain.
Specifically, when the user specifies, by operating the operation management client 10, a storage domain and a storage device registered on the storage domain, the system resource manager 21 receives the information from the operation management client 10.
The system resource manager 21 reads the inter-server/storage-domain link data 680 shown in
Moreover, the system resource manager 21 checks uniformity of wire connections between servers that belong to the specified server domain and storage devices that belong to the storage domain, in corporation with the storage RM 25 and the storage RM agent 115a.
In addition, a wire connection between the FC switches 750a and 750b and a RAID device 760a that belongs to the storage domain 740, and a wire connection between the FC switches 750a and 750b and a RAID device 760b that belongs to the storage domain 740 are uniform.
The system resource manager 21 performs the uniformity check of the above wire connections based on information on WWPN (world wide port name). At this check, the system resource manager 21 reads information on redundancy of a storage domain path from the storage domain data 360 shown in
In
The RAID device WWPN data 770a and 770b contains information on CA (channel adapter) and WWPN. The CA is information that identifies a channel adapter included in the RAID devices 760a and 760b. The WWPN is information on WWPN assigned to the channel adapter included in the RAID devices 760a and 760b.
The FC switch WWPN data 780a and 780b contains information on port and other side WWPN. The port is information that identifies ports of the FC switches 750a and 750b. The other side WWPN is information on WWPN assigned to the channel adapter of the RAID devices 760a and 760b connected to the ports of the FC switches 750a and 750b, or information on WWPN assigned to an HBA (host bus adapter) of the servers 730a and 730b connected to the ports of the FC switches 750a and 750b.
The server WWPN data 790a and 790b contain information on HBA and WWPN. The HBA is information that identifies an HBA included in the servers 730a and 730b. The WWPN is information on WWPN assigned to the HBA included in the servers 730a and 730b.
The system resource manager 21 can check the uniformity of a wire connection between devices by collecting the RAID device WWPN data 770a and 770b, the FC switch WWPN data 780a and 780b, and the server WWPN data 790a and 790b from the RAID devices 760a and 760b, the FC switches 750a and 750b, and the servers 730a and 730b, and by checking the associations of WWPN.
After that, the system resource manager 21 registers a storage area having an LUN (logical unit) set in advance and a storage area having an unset LUN, as storages for a pool.
Subsequently, the system resource manager 21 performs processing of creating a server group (step S104). The processing performed at step S104 are explained in detail below.
The system resource manager 21 receives information on a storage template set by the user by operating the operation management client 10, and registers the information on the system resource DB 26 as storage template data 800 described below. The storage template is setting information on configuration of later created storage for server group.
The storage template is identification information that identifies an established storage template. The disk type is information on a type of an application of the disk included in the storage template.
For example, “root” denotes that the disk is used to store therein system data, “local” denotes that the disk is used to store therein individual server data, and “shared” denotes that the disk is used to store therein shared data among servers.
The disk name is a name that identifies a disk, and assigned to each of the disks. The reliability need is information on a reliability needed for the disk. The load level is information on a load level on the disk. The disk capacity is a storage capacity of the disk. The boot disk is information on whether the disk is used to boot a system.
Subsequently, the system resource manager 21 receives information on server groups that are set by a user by operating the operation management client 10, and stores the information in the system resource DB 26 as a server group data 810 explained below.
The server group is identification information that identifies a group, if servers included in the same server domain are classified into one or a plurality of groups. The server domain is information on a server domain to which server groups belong. The software distribution image is information that identifies an image file in software distributed to the server that belongs to the server group.
The version is information on a version of a software distribution image. The storage template is same information as the storage template explained in connection with
The system resource manager 21 registers information on a storage group corresponding to the server group on the system resource DB 26 as a server/storage group link data 820. In this case, the storage group denotes that storages included in the same storage domain are classified into one or a plurality of groups.
The server group is the same information as the server group shown in
At the time of creating the storage group, the system resource manager 21 retrieves information on a storage template associated with the server group from the server group data 810 shown in
The system resource manager 21 creates the storage group with respect to each of disk types such as “root”, “local”, and “shared” with respect to each of the server groups, and registers the information on the server/storage group link data 820.
Furthermore, the system resource manager 21 retrieves information on a storage domain corresponding to the server domain to which the server group belongs from the inter-server/storage-domain link data shown in
After that, the system resource manager 21 transmits a command for causing the AP managing unit 116a to recognize that the server group is added to the AP managing unit 116a. Specifically, the system resource manager 21 transmits “issvgrp add” shown in
Subsequently, the system resource manager 21 receives information on correspondence relations among the server groups that are set by the user by operating the operation management client 10, and registers the information on the system resource DB 26 as inter-server-group link data 830 explained below.
The front server group is information on a server group closer to the Internet 70 among server groups that are linked via the network group. In this case, the network group denotes a network group consisting of a combination of the network sub groups that link the server groups as explained in connection with
The network group is identification information that identifies the network group. The back server group is information on a server group located more distant from the Internet 70 among server groups that are linked via the network group.
The system resource manager 21 stores information on the network group in the system resource DB 26 as network group data 850 explained below.
Specifically, the system resource manager 21 retrieves the inter-server-domain link data 670 shown in
The system resource manager 21 retrieves the network domain data 480 shown in
Furthermore, the system resource manager 21 retrieves the network sub group data 660 shown in
Subsequently, the system resource manager 21 classifies a network device corresponding to the information on a device retrieved from the network domain data 480 shown in
The load distribution group is information that identifies a group, if a load balancer is classified into one or a plurality of groups. The load balancer name is a name that identifies a load balancer. The representative IP is information on an IP address assigned to each of the load distribution groups.
The system resource manager 21 creates information on correspondence relations among the network groups based on information on configurations of the network domain, the network sub group, the load distribution group, and the like, which belong to each of the network groups, and then stores the information in the system resource DB 26 as the network group data 850 explained below.
The network group is the same information as the network group explained in connection with
The front network sub group corresponds to the network sub group explained in connection with
The load distribution group is the same information as the load distribution group explained in connection with
Furthermore, the system resource manager 21 sets up VLAN of the network sub group in a switch that is registered on the network sub domain data 470 shown in FIG. 13 in cooperation with the network RM 24 and the network sub RM 54.
Subsequently, the system resource manager 21 adds a first server into the server group, and performs a process of creating a software image of software that is installed on the server (step S105). The process at step S105 is explained in detail below.
First, if the user specifies a server and a server group where the server is registered by operating the operation management client 10, the system resource manager 21 receives information on the server and the server group and registers the server on the server group.
The system resource manager 21 retrieves the server group data 810 shown in
The storage RM 25 sets up such a logical volume that fulfills the setting condition of the storage template obtained by the system resource manager 21 to a pooled storage, and performs a process of assigning the storage in which the logical volume is set to the server group.
In the example shown in
Referring back to
The RAID-level setting data 940 contains information on reliability need, load level, and RAID level. The reliability need is the same information as the reliability need explained in connection with
Referring back to
The RAID device data 950 contains information on total amount of required disk capacity, RAID device model, data access speed, the number of disk drives configuring RAID group (in a case of RAID0+1), the number of disk drives configuring RAID group (in a case of RAID5), and the maximum number of RAID groups.
The total amount of required disk capacity is information on a total amount of a disk capacity required for a logical volume. The RAID device model is information on a model of a RAID device appropriate for ensuring the total amount of the required disk capacity.
The data access speed is information on a data access speed of a disk drive that is specified by the RAID device model. The data access speed contains information on three types of the “first”, “second”, and “third” disk drives in order of data access speed.
The number of disk drives configuring RAID group (in a case of RAID0+1) is information on the number of disk drives configuring a RAID group in the case of RAID0+1. The number of disk drives configuring RAID group (in a case of RAID5) is information on the number of disk drives configuring a RAID group in the case of RAID5. The maximum number of RAID groups is information on the maximum number of RAID groups to be created.
Referring back to
In this case, the unique information is information on a type of the “first” disk drive out of the data access speed, the number of disk drives configuring the RAID group (in the case of RAID0+1), the number of disk drives configuring the RAID group (in the case of RAID5), and the maximum number of the RAID groups.
Then, the storage RM 25 creates a logical volume (step S205). Specifically, the storage RM 25 creates such a logical volume that fulfills each of the necessary conditions of the logical volume, and sets up the logical volume in the RAID device.
On the logical-volume-configuration output screen 880 shown in
Referring back to
In the example shown in
Upon creating the RAID groups, the storage RM 25 sets disk drives that belong to each of the RAID groups to disk drive types that are determined based on the data access speed of the RAID device data 950 shown in
Furthermore, the storage RM 25 sets the number of disk drives that configure a RAID device to the number of disk drives determined based either on the number of disk drives configuring the RAID group (in the case of RAID0+1) or the number of disk drives configuring the RAID group (in the case of RAID5) those included in the RAID device data 950 shown in
Furthermore, the storage RM 25 creates such RAID groups that the number of the RAID groups is below the maximum number of the RAID groups included in the RAID device data 950 shown in
On the logical-volume-configuration output screen 880 shown in
Referring back to
After that, the system resource manager 21 registers the logical volume set in the RAID device as a storage sub group by associating the logical volume with server groups to which each of servers belongs, and sets an access right to the server groups of the servers. Specifically, the system resource manager 21 stores information on server group, storage sub group, and availability of access in the provisioning configuration data 710 shown in
When the logical volume configured in the RAID device is to be recognized by a server and registered as a storage sub group, the storage RM 25 sets up the logical volume in the following procedure.
In this case, the affinity group is information on a correspondence relation between a logical unit number (LUN) to be recognized by the server and a logical volume (LV) number in the RAID device.
Moreover, in
The logical volumes LV0 and LV1 in the RAID device α, which are added into the storage group 990, are set to belong to an affinity group 0 and an affinity group 1. Also, the logical volumes LV12 and LV13 in the RAID device β are set to belong to an affinity group 10 and an affinity group 11.
The RAID device is identification information that identifies each of RAID devices. The affinity group is information on an affinity group that is set in each of the RAID devices. The LUN is identification information that identifies a logical volume when accessed from the server A or the server B. The LV is identification information that identifies a logical volume.
Referring back to
The storage RM 25 sets a multipath for a logical unit (step S303).
The multipath configuration data 1020 contains information on multipath instance and LUN. The multipath instance is information that identifies an instance of set multipath. The LUN is information that identifies a logical unit that corresponds to the set multipath instance and that is to be recognized by either the server A or the server B.
The storage RM 25 registers the set multipath instance as a constituent element of the mirror volume on a cluster resource of the server to which a clustering is performed (step S304). Thereafter, the storage RM 25 sets a mirror volume group constituted of a pair of volumes of different RAID devices, using the multipath instance registered on the cluster resource (step S305).
An intra-server storage configuration 1000 set inside the server “A” or the server “B” is described in
The mirror volume is identification information that identifies a set mirror volume. The configuring disk is identification information that identifies a logical unit constituting the mirror volume. The configuring disk stores therein information on the multipath instance stored as the multipath configuration data 1020 shown in
The affinity group data 1010 shown in
Referring back to the explanation of the processing of creating the software image at step S105 shown in
More specifically, the network RM 24 reads information on a network group that includes the server group to which the server is added as a front server group and a back server group, from the inter-server-group link data 830.
The network RM 24 reads the network group data 850 shown in
Subsequently, the network RM 24 reads the network sub group data 660 shown in
The IP address management data 1040 contains information on IP address and assignment destination. The IP address is information on an IP address assigned to the server. The assignment destination is information that identifies the server to which the IP address is assigned.
Subsequently, the network RM 24 assigns a load distribution group including a representative IP address to the network group corresponding to the server group to which the server is added, based on the load distribution group data 840 shown in
A user installs the software such as OS to be installed in the server to the storage sub group associated with the server to be added to the server group. The storage sub group is configured by using a SAN technology.
After the installation is finished, the software sub RM 53 creates the software image constituted of a group of the software such as OS, device driver, and application software, in cooperation with the software RM 23 and the software RM agent 113a, and stores the software image created in the domain resource DB 55.
More specifically, the software RM 23 reads the middleware cooperation IF data 330 shown in
In other words, the software RM agent 113a transmits a command for stopping the function of the AP managing unit 116a and stops the function of the AP managing unit 116a. The software sub RM 53 shutdowns the system of the server. The software sub RM 53 performs a network boot of the server using tentative OS stored in the domain resource DB 55 of the domain management server 50 of the server.
Thereafter, the software sub RM 53 creates the software image of the software installed in the server started. The software RM 23 registers information on the software image on the system resource DB 26 as software image management data 1050 described below.
The software image name is a name of a software image. The format is information that indicates whether a software image is created in archive format or in patch format. The OS property is information that indicates whether a software image is an OS software image. The software name is a name of software for which a software image is created.
The software sub RM 53 creates a software distribution image to be distributed to other servers, based on the software image created. Specifically, the software sub RM 53 creates a software distribution image in which software images are grouped of a plurality of software installed in a storage for the first server.
The system resource manager 21 stores information on the software distribution image in the system resource DB 26 as software distribution image management data 1060 described below.
The software distribution image name is a name of a software distribution image. The version is a version of the software distribution image. The server architecture is a CPU architecture of the server to which the software distribution image is distributed. The software image/snapshot is information that indicates a software image or a snapshot included in the software distribution image.
The snapshot is a software image of the software installed in the server at a specific timing. The system resource manager 21 registers information on the snapshot on the system resource DB 26 as snapshot management data 1070 described below.
The software RM 23 reads the middleware cooperation IF data 330 shown in
More specifically, the software RM agent 113a transmits a command for starting the AP managing unit 116a on standby and starts the AP managing unit 116a. The network RM 24 connects the server to a VLAN by performing a setting of the VLAN to a switch, activates a load distribution function of the load balancer, and assigns the server as a target server to which a load is distributed.
Thereafter, the system resource manager 21 reads the middleware cooperation IF data 330 shown in
More specifically, the system resource manager 21 transmits a command that enables a recognition of an addition of the server group to the AP-management control unit 27. The AP-management control unit 27 performs an installation and a setting of application programs to the server in cooperation with the AP managing unit 116a, and sets the server to be in a state available for a task.
Referring back to
The system resource manager 21 registers the server on the server group (step S402). Subsequently, the system resource manager 21 reads the management target server data 700 shown in
When the server architecture of the server is enabled for the installation of the software image (YES at step S403), the storage RM 25 performs a processing of setting the storage to the server, in the same manner for setting the storage to the first server (step S404). Specifically, the storage RM 25 performs a processing of setting the logical volume described in connection with
Thereafter, the network RM 24 performs a network boot of the server registered on the server group using a tentative OS in the same manner for setting the network to the first server, and performs a setting of the network to the server (step S405).
The software sub RM 53 expands the software distribution image created from the software installed in the first server onto the storage sub group associated with the second server, and reboots the server using expanded software (step S406).
When the software distribution image is expanded onto the storage sub group associated with the server, the software RM 23 stores information on the software distribution image distributed in the system resource DB 26.
The server is information that identifies a server to which the storage sub group is assigned. The storage sub group is information that identifies a storage sub group on which the software distribution image is expanded. The software distribution image is information on a software distribution image expanded on the storage sub group. The status is information that indicates a status of distributing the software distribution image.
Referring back to
Specifically, the network RM 24 assigns an IP address to the second server based on information on the subnet to which the first server belongs, at the timing of rebooting the server. The information on the IP address assigned to the second server is stored in the IP address management data 1040 shown in
Thereafter, the network RM 24 connects the server to VLAN by performing a setting of VLAN to a switch, and registers the server on the load balancer as a target server to which a load is distributed.
The system resource manager 21 transmits a command for causing the AP-management control unit 27 to recognize that the server is added to the server group to the AP-management control unit 27. The AP-management control unit 27 performs an installation or a setting of the application to the server in cooperation with the AP managing unit 116a, and sets the server to be in a state available for the task.
When adding the third server or subsequent servers to the server group, the processing of adding the server described in connection with
A processing of deleting the server added to the server group from the server group is described below.
As shown in
Subsequently, the network RM 24 returns the IP address assigned to the server (step S503). The software sub RM 53 reboots the server by a network boot, using the tentative OS stored in the domain resource DB 55 of the domain management server 50 (step S504).
The storage RM 25 removes a disk assigned to the server to be deleted from the server group (step S505). The storage RM 25 changes a SAN zoning that is a logical connection relation between the servers and the storages set to the servers, and sets a SAN zoning between the servers excluding the server and the storages (step S506).
A processing is described, of previously setting information on software, network, and the like to a server in a pool for adding the server resource at a high speed when abnormality occurs in the server or a load on the server increases.
The system resource manager 21 reads the server/storage group link data 820 shown in
Thereafter, the system resource manager 21 reads the distribution management data 1080 shown in
The network RM 24 performs a network setting of the server (step S608). The server RM 22 starts the server (step S609), and the system resource manager 21 transits the server to the pool (step S610).
The server RM 22 stops the server (step S611). The network RM disconnects the server from the network (step S612). When the server is transited to a next server group (YES at step S613), the system resource manager 21 transits the server to the next server group (step S614) and the process control returns to step S602. On the other hand, when the server is not transited to the next server group (NO at step S613), the process ends.
As shown in
The system resource manager 21 reads the server/storage group link data 820 shown in
Subsequently, the system resource manager 21 reads the distribution management data 1080 shown in
The system resource manager 21 reads the server group data 810 shown in
The network RM 24 performs a network setting of host5 (step S708), the server RM 22 starts host5 (step S709), and the system resource manager 21 transits host5 to the pool (step S710).
Thereafter, the server RM 22 stops host5 (step S711), the network RM 24 disconnects host5 from the network (step S712), and the system resource manager 21 transits host5 to the B_Web server Group (step S713).
The system resource manager 21 reads the server/storage group link data 820 shown in
Subsequently, the system resource manager 21 reads the distribution management data 1080 shown in
The system resource manager 21 reads the server group data 810 shown in
The network RM 24 performs a network setting of host5 (step S720). The server RM 22 starts host5 (step S721). The system resource manager 21 transits host5 to the pool (step S722). The server RM 22 stops host5 (step S723). The network RM 24 disconnects host5 from the network (step S724).
A processing of adding server resource when abnormality occurs in the server or a load on the server increases is explained below.
The system resource manager 21 reads the server/storage group link data 820 shown in
Upon reading the server group data 810 shown in
The network RM 24 resumes distribution of processing by the load balancer to the server (step S809), the server RM starts the server (step S810). Although the processing is described in the example shown in
A processing of adding the server resource when abnormality occurs in the server or a load on the server increases shown in
As shown in
The system resource manager 21 reads the server/storage group link data 820 shown in
The system resource manager 21 reads the server group data 810 and acquires information on the software of the A_Web server Group (in this flowchart, it is assumed to receive information indicating that the software of the A_Web server Group is A_OS_Web_image version 1.1) (step S905).
Because the software of the A_Web server Group is updated from version 1.0 to version 1.1, the network RM 24 stops distribution of processing by the load balancer to host5 (step S906). The system resource manager 21 reads the software distribution image management data 1060 shown in
The network RM 24 connects host5 to the network (step S908), and resumes distribution of processing by the load balancer to the server (step S909). The server RM 22 starts the server (step S910).
In this manner, because settings of software or a network corresponding to the server group to which the server is added is performed in advance for the servers in the pools, it is possible to add the server in the pools and to restore the operation of the server group in an expedited manner.
Next, various screens are explained, which are displayed on the operation management client 10 through a resource assignment management processing by the system resource manager 21.
As shown in
In addition, when the pooled server is specified on the server list screen 1160 by a user by operating a mouse and the like, and if an add button is clicked, an execution request for an processing of adding the specified server to the server group is transmitted to the system resource manager 21 and the processing of adding the server is performed.
When the server that belongs to the sever group is specified on the server list screen 1160 by a user by operating a mouse and the like, and if a delete button is clicked, a deletion request for an processing of deleting the specified server from the server group is transmitted to the system resource manager 21 and the processing of deleting the server is performed.
When the pooled storage is specified on the storage list screen 1170 by a user by operating a mouse and the like, and if an add button is clicked, an execution request for an processing of adding the specified storage to the storage group is transmitted to the system resource manager 21 and the processing of adding the storage is performed.
When the storage that belongs to the storage group is specified on the storage list screen 1170 by a user by operating a mouse and the like, and if a delete button is clicked, a deletion request for an processing of deleting the specified storage from the storage group is transmitted to the system resource manager 21 and the processing of deleting the storage is performed.
The various processings described in the above embodiment can be realized by causing a computer to execute a pre-prepared programs. An example of the computer that executes a resource-change processing program is explained below in connection with FIGS. 57 to 59.
The HDD 1260 stores therein programs that realize functions same as those of the site management server 20, that is, a system-resource-change processing program 1260b and an AP management control program 1260c shown in
The system-resource-change processing program 1260b and the AP management control program 1260c can be integrated or be distributed for a recording as appropriate.
When the CPU 1280 reads and executes the system-resource-change processing program 1260b and the AP management control program 1260c from the HDD 1260, they work as functions of a system-resource-change processing process 1280a and an AP management control process 1280b.
The system-resource-change processing process 1280a corresponds to the system resource manager 21, the server RM 22, the software RM 23, the network RM 24, and the storage RM 25 shown in
The HDD 1260 also stores therein system resource data 1260a. The system resource data 1260a corresponds to various data stored in the system resource DB 26 shown in
The CPU 1280 stores therein various data related to a management of the resource as the system resource data 1260a, reads the system resource data 1260a from the HDD 1260 to store read data in the RAM 1270, and performs various data processings based on system resource data 1270a stored in the RAM 1270.
The HDD 1360 stores therein programs that realize functions same as those of the domain management servers 50 and 60, that is, a domain-resource-change processing program 1360b shown in
When the CPU 1380 reads and executes the domain-resource-change processing program 1360b from the HDD 1360, it works as a function of a domain-resource-change processing process 1380a.
The domain-resource-change processing process 1380a corresponds to the system resource domain manager 51, the server sub RM 52, the software sub RM 53, and the network sub RM 54.
The HDD 1360 also stores therein domain resource data 1360a. The domain resource data 1360a corresponds to various data stored in the domain resource DB 55 shown in
The CPU 1380 stores therein various data related to a management of the resource in the domain as the domain resource data 1360a, reads the domain resource data 1360a from the HDD 1360 to store read data to the RAM 1370, and performs various data processings based on the domain resource data 1370a stored in the RAM 1370.
The HDD 1470 stores therein programs that realize same functions as those of the server 110a, that is, an agent-resource-change processing program 1470a and an AP management program 1470b. The agent-resource-change processing program 1470a and the AP management program 1470b can be integrated or be distributed for a recording as appropriate.
When the CPU 1480 reads and executes the agent-resource-change processing program 1470a and the AP management program 1470b from the HDD 1460, they work as functions of an agent-resource-change processing process 1480a and an AP management process 1480b.
The agent-resource-change processing process 1480a corresponds to the resource manager agent 111a, the server RM agent 112a, the software RM agent 113a, the network RM agent 114a, and the storage RM agent 115a shown in
The system-resource-change processing program 1260b, the AP management control program 1260c, the domain-resource-change processing program 1360b, the agent-resource-change processing program 1470a, and the AP management program 1470b are not necessary to be recorded in advance in the HDD 1260, the HDD 1360, or the HDD 1470.
For example, it is acceptable to store each of programs in a removable physical medium, such as a flexible disk (FD), CD-ROM, an MO disk, a DVD disk, an optical magnetic disk, or an IC card, to be inserted to the computer 1200, 1300, or 1400. It is also acceptable to store the programs in a built-in physical medium, such as an HDD installed in a computer or attached to an outside of the computer, or in other computers (or a server) connected to the computer 1200, 1300, or 1400 via a public line, the Internet, a LAN, or a WAN. In this case, it is possible for the computer 1200, 1300, or 1400 to read each of the programs and to execute the programs.
As described above, according to the embodiment, the software RM 23 and the network RM 24 perform in advance to the server in the pool, an installation or a setting of the software corresponding to the server group to which the server is added, using information in the system resource DB 26. Therefore, it is possible to restore the server group at low costs and at high processing speed by adding the server to which the settings of the software and a network are previously performed, when a failure occurs in the server included in the server group or load on the server group increases.
As described above, according to one aspect of the present invention, a plurality of server groups is configured with a plurality of servers, setting information as information on settings of software, network, and storage for the server group is acquired, settings corresponding to the server group is performed in advance to a backup server based on the setting information, and the backup server to which the setting has been performed is added to the server group when abnormality occurs in the server group or load on the server group increases. Therefore, it is possible to restore the server group in an expedited manner.
Although the invention has been described with respect to a specific embodiment for a complete and clear disclosure, the appended claims are not to be thus limited but are to be construed as embodying all modifications and alternative constructions that may occur to one skilled in the art that fairly fall within the basic teaching herein set forth.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP04/15036 | Oct 2004 | US |
Child | 11786440 | Apr 2007 | US |