The invention relates to a method and mechanism for performing rolling software upgrades to a distributed computing system.
Over time, many types of software applications will undergo some sort of change. These changes may occur for a variety of reasons. For example, given the complexity of modern software applications, it is well known that most software contains coding errors or “bugs” that will need correction. One reason to upgrade from an earlier version to a later version of a software application is to correct errors or bugs that may exist in the earlier version. Another reason for changing a software application is to introduce improvements to the operation or functionality of the software application.
A “rolling upgrade” refers to the process of performing software upgrades to a live existing software installation in a distributed environment in which the individual instances, nodes, or entities of the distributed system (referred to herein as “members”) are upgraded in a staggered manner. This form of upgrade ensures availability of the application during software upgrades, and thus minimizes or eliminates planned downtime while contributing to high availability goals. As used herein, the term member may encompass either a single instance/node/entity or a collection of such instances/nodes/entities.
At each member, there are numerous ways to upgrade a software application from an earlier version to a later version. A common approach is for a software developer to create patches and patch sets that are applied to a copy of the software binary or executable. Another common approach is to create a new object having the same location reference. Tools are often provided to perform the software upgrades or installations.
Performing an upgrade or change to an existing software application typically requires a shutdown of either/both the member or software enterprise. For example the upgrade can be performed by shutting down the software, implementing the upgrade, and then bringing the member back up so that availability is restored.
With modern software, it can be anticipated that software developers will provide upgrades and changes on an ongoing basis. In fact, many IT (“information technology”) departments will periodically schedule planned events to perform upgrades to their software installations. These events could result in significant planned downtimes. It is desirable to limit the effects of these downtimes as much as possible since they could affect the availability of mission critical systems, potentially resulting in productivity and financial losses for organizations.
If the system being upgraded is a distributed system having multiple independent members where the software is located in the members' local directories, then in one approach, the upgrade can be performed individually at each member so that other members do not suffer downtime while the affected member is being upgraded. However, problems arise with this approach if it is implemented in networked and shared filesystem environments in which multiple members operate with the same shared software installations. Some examples of this type of configuration are when multiple members access the same software installation at a shared filesystem using the NFS (network file system) mechanism or in the Cluster File System such as the Oracle Cluster File System (OCFS) available from Oracle Corporation of Redwood Shores, Calif. With this type of architecture, since the application files are shared, performing a rolling upgrade could result in all members being shutdown during the upgrade process, resulting in total unavailability for the systems during the downtime.
For operating system (OS) upgrades, one approach for handling this is provided in the Tru64/TruCluster system which offers OS level support to perform rolling upgrades on their Cluster File System. The TruCluster model uses tagged files and kernel parameters to support multi-versioning and version switching. However, this approach may result in inefficient performance involving 2n−1 reboots to the networked members when performing the OS upgrades where n is the number of members being upgraded.
Therefore, to address these and other problems, what is described herein is an improved method and mechanism for performing rolling upgrades, e.g., to shared software installations in a distributed environment. The present approach eliminates or minimizes extraneous downtime when performing a rolling upgrade, thereby improving performance and availability for users of the shared software installation. In one embodiment, a rolling upgrade is performed by defining a private symbolic link for each member that is upgraded to reference the upgraded version of the shared software installation. This approach can be performed upon any computing system, whether single node (e.g., a multi-instance application on a single computer) or a multi-node system (e.g., a cluster or network of stations).
Further details of aspects, objects, and advantages of the invention are described below in the detailed description, drawings, and claims.
The accompanying drawings are included to provide a further understanding of the invention and, together with the Detailed Description, serve to explain the principles of the invention. The same or similar elements in the figures may be referenced using the same reference numbers.
The present invention provides a method and mechanism for performing a rolling upgrade to shared software installations in a distributed environment, e.g. for a networked, shared filesystems. The present approach eliminates or minimizes extraneous downtime when performing a rolling upgrade, thereby improving performance and availability for users of the shared software installation.
In this type of architecture, performing a rolling upgrade could cause significant downtime in the system 100. This is because one has to stop all the instances of the application running the distributed members 1, 2, 3 to change or upgrade the installed software in the shared files 106. In other words, none of the clients 102 will be able to perform work in the system while an upgrade is being performed in this approach, since all members that require access to shared files 106 would be down.
Embodiments of the present invention provide an improved method and mechanism for performing rolling upgrades that do not require all members using the shared file system to be brought down when upgrading or changing installed software applications. In one embodiment, the rolling upgrade is performed using private symbolic links (PSL).
At this point, it is helpful to generally describe a symbolic link. A symbolic link is a logical reference from a first file/pathname to a second file/pathname, which is most commonly used in UNIX-based systems. Most operations (e.g., read and execute operations) that are called to operate upon the symbolic link are instead redirected to operate upon the filename referenced by the symbolic link.
In a present embodiment, a private symbolic link mechanism is used to perform a rolling upgrade. Unlike a standard symbolic link, a PSL does not provide a universally visible symbolic link between two file/pathnames. Instead, the PSL makes the link between two filenames visible only to members specified or authorized for the PSL. The present disclosure defines the semantics of a PSL that is employed in one embodiment of the invention for performing rolling upgrades.
It is desired to establish a configuration of the system such that different entities in the system will symbolically link from the same pathname/filename to different files in the shared file system. This is accomplished in the present embodiment by establishing one or more private symbolic links in which a link criteria or link configuration will determine which file will be referenced by the symbolic link for particular entities in the system. As used herein, such entities include any object, member, application or individual which is capable of using or accessing a shared file. For the purposes of illustration, the term “member” will be used instead of “entities” in the rest of this description.
An example structure 220 is shown in
In structure 220, a first entry 222 has been configured to include a link criteria definition that applies to all members having a given software application at version 1. This means that the PSL associated with entry 222 only applies to a member only if the member has the stated software application at version 1. If the member is running the software application at any other version number, then the PSL defined by entry 222 does not apply to that member. Entry 222 is configured such that it “link from” a filename/pathname for “/binaries/version”. The “link to” information for entry 222 has been defined as “/binaries/version1”. This means that any member that is a member of the group defined by the link criteria information for entry 222 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version1” file/pathname.
Similarly, it can be seen that entry 224 has link criteria information that defines a PSL which applies to members running the software application at version 2. Entry 224 is configured such that it “link from” a filename/pathname for “/binaries/version”. The “link to” information for entry 224 has been defined as “/binaries/version2”. This means that any member that is a member of the group defined by the link criteria information for entry 224 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version2” file/pathname. Entry 226 has link criteria information that defines a PSL which applies to members running the software application at version 3. Entry 226 is configured such that it “link from” a filename/pathname for “/binaries/version”. The “link to” information for entry 226 has been defined as “binaries/version3”. This means that any member that is a member of the group defined by the link criteria information for entry 226 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version3” file/pathname.
In the example of
One specific type/variant of a PSL that can be created is a member private symbolic link (MPSL). The MPSL creates a symbolic link that only applies to the member defined for the MPSL. Therefore, the member of the group defined for a MPSL may be limited to a single identified member.
Entry 252 in structure 250 has the name “member 1” defined in the member column. This means that the PSL associated with entry 252 only applies to the member identified as “member 1”. Entry 252 is configured such that it “link from” a filename/pathname for “/binaries/version”. The “link to” information for entry 252 has been defined as “/binaries/version1”. This means that member 1 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version1” file/pathname. Similarly, it can be seen that entry 254 has been defined to only apply to the member named “member 2”. Entry 254 is configured such that it “link from” a filename/pathname for “/binaries/version”. The “link to” information for entry 254 has been defined as “/binaries/version2”. This means that member 2 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version2” file/pathname. Entry 256 has been defined to only apply to the member named “member 3”. Entry 256 is configured such that it “link from” a filename/pathname for “binaries/version”. The “link to” information for entry 256 has been defined as “/binaries/version3”. This means that member 3 will have a private symbolic link from the “/binaries/version” file/pathname to the “/binaries/version3” file/pathname.
In the examples of
The private symbolic link can be used to implement a process for performing rolling upgrades to software.
In the approach of
At 334, links are configured such that members that are to be upgraded to the new version of the software will point to the newly created copy of the new version of the software. For a rolling upgrade, this process of pointing the members will occur in a staggered manner such that availability to at least one or more members is retained throughout the upgrade process. A determination is made whether there are any additional members to process (335). If so, then action 334 is repeated for each additional member to create a link to the new version. At 336, the links are deleted after the members point to the new version of the software.
In an embodiment, the MPSL mechanism is used to providing this linking capability. With this approach, a member is identified for which the rolling upgrade is desired. During the upgrade, the member and/or its running software application is brought down. It is noted that in this approach, only the specific member presently identified for the rolling upgrade is brought down. All other members still operating against the old version of the software can continue running without interruption. After the member has been brought down, a private symbolic link is created for the identified member. The private symbolic link creates a symbolic link between the file/pathname of the old version and the file/pathname of the new version of the software. The private symbolic link only applies to the members of the group associated with the MPSL. In one embodiment, the private symbolic link is a member private symbolic link, and the only member of the group associated with the MPSL is the identified member. After linking occurs, the member is brought back up. Because of the MPSL, the software application will be brought up automatically referencing the application files from the new version of the software. A determination is made whether additional members need to be upgraded during the rolling upgrade. If so, the process is repeated until all members that need to be upgraded have been upgraded.
Once all relevant members have been upgraded, a determination is made whether to remove the old version of the software (338). There are many reasons not to remove the old software. For example, it is possible that there is still a reason to continue using the old software version on some of the members in the network. If, however, the decision has been that the old software version is no longer needed, then the old software version is removed (340).
In the embodiment of
At 322, a member creates a private copy of a new version of the software. The private copy is created in the shared file system without removing the old version of the software. Any appropriate approach can be taken to create the new private copy. The newly created copy of the new version of the software co-exists in the shared file system with the old version of the software, but having a different pathname or filename from the old version.
In the approach of
At 324, a link is configured such that the member that is to be upgraded will point to the newly created private copy of the new version of the software. For a rolling upgrade, this process of pointing the members will occur in a staggered manner such that availability to at least one or more members is retained throughout the upgrade process. The MPSL mechanism can be used to providing this linking capability. After linking occurs, the member is brought back up. Because of the MPSL, the software application will be brought up automatically referencing the application files from the new version of the software.
A determination is made whether there are any additional members to upgrade (323). If so, then another private copy of the new version is created (322) and a link is created to the new private copy (324). The process is repeated until all members that need to be upgraded have been upgraded. At 326, the link(s) are deleted after the member(s) point to the new version of the software. At a later point, the members can be configured to access the same copy of the new version of the software, rather than their own private copies.
Once all relevant members have been upgraded, a determination is made whether to remove the old version of the software (328). If the decision has been that the old software version is no longer needed, then the old software version is removed (330).
To illustrate the embodiment of the invention of
Moving to
At this point, the first member to upgrade is identified. Assume that the first member to upgrade is member 1; Referring to
A private symbolic link is created to associate member 1 with the new version 406b of the software. This is shown in
Referring to
At this point, it can be seen that members running both the old and new versions of the same software application are simultaneously running the system. In addition, it is noted that when performing the rolling upgrade, the process can be configured such that downtime only occurs for a single member at a time. However, multiple members can still be simultaneously shut down for upgrades within the scope of the invention. The advantage of this approach over the prior approaches is that less than all of the members need be brought down when performing a rolling upgrade for shared application files.
The next action is to upgrade member 2, which is still operating using the old version 406a of the software. Referring to
Referring to
Since all members have now been upgraded, it is now possible to remove the old version 406a of the software and to remove the links. Turning to
The process of
Moving to
At this point, the first member to upgrade is identified. Assume that the first member to upgrade is member 1. Referring to
A private symbolic link is created to associate member 1 with the new version 506b of the software. This is shown in
Referring to
The next action is to upgrade member 2, which is still operating using the old version 506a of the software. Another private copy 506c of the new version of the software is created. Referring to
Referring to
At this point, the system is configured such that all future requests to access the software at the specified reference location will access either private copy 506b or 506c of the new version of the software. Since all members have now been upgraded, it is now possible to upgrade the old version 506a of the software to the new version. Turning to
Referring to
Since the members 1 and 2 no longer link to the private copies 506b and 506c of the software, these private copies can be removed.
Multiple versions of the software application can co-exist in the distributed system through the rolling upgrade process. Since member private symbolic links are used, each member may point to different versions of the software. Therefore, if it is desired to leave individual members at different versions of the software, this can be accomplished without interfering with the ability of other members to point to other versions of the software.
The present approach can be applied not only to distributed systems in which the members are on different network nodes, but can also be applied to perform rolling upgrades in which multiple entities occupy the same network node. This type of situation may occur, for example, for a multi-instance application that resided on a single node. The present approach can be used to ensure a high level of availability for the multiple instances on the single network node during a rolling upgrade be creating private symbolic links for the individual instances, as described with respect to
Therefore, what has been described is an improved process for performing a rolling upgrade to software. It is noted, however, the principles described herein are applicable to a wide variety of different applications in which file branching is desired. For example, the present application may be applicable to situations in which the shared files are data files, rather than shared application executables. This occurs, for example, with stored web pages at a central server. Consider if a first member operate a first type of web browser and a second member operates a second type of web browser, in which it is desirable to maintain a different version of the same web page for each type of browser. In this situation, a private symbolic link could be used to automatically allow each member to correctly reference the specific web page corresponding to its browser type, even though both members may actually attempt to access the same file/pathname or web page name.
When the software to access the shared data files are upgraded, the above process can be used to provide seamless changes to the specific data files accessed by the upgraded applications. Again using the web browser application as an example, consider a browser application that is changed from an earlier version to a later version on a network member. Assume that it is desirable to maintain a different version of the same web page for each version of browser. When the member upgrades to the new version of the browser, a private symbolic link can be defined to allow the upgraded to member to automatically access the correct web page after the upgrade.
The execution of the sequences of instructions required to practice the invention may be performed in embodiments of the invention by a computer system 1400 as shown in
Each computer system 1400 may include a communication interface 1414 coupled to the bus 1406. The communication interface 1414 provides two-way communication between computer systems 1400. The communication interface 1414 of a respective computer system 1400 transmits and receives signals, e.g., electrical, electromagnetic or optical signals, that include data streams representing various types of information, e.g., instructions, messages and data. A communication link 1415 links one computer system 1400 with another computer system 1400. A computer system 1400 may transmit and receive messages, data, and instructions, including program, i.e., application, code, through its respective communication link 1415 and communication interface 1414. Received program code may be executed by the respective processor(s) 1407 as it is received, and/or stored in the storage device 1410, or other associated non-volatile media, for later execution.
In an embodiment, the computer system 1400 operates in conjunction with a data storage system 1431, e.g., a data storage system 1431 that contains a database 1432 that is readily accessible by the computer system 1400. The computer system 1400 communicates with the data storage system 1431 through a data interface 1433. A data interface 1433, which is coupled to the bus 1406, transmits and receives signals, e.g., electrical, electromagnetic or optical signals, that include data streams representing various types of signal information, e.g., instructions, messages and data. In embodiments of the invention, the functions of the data interface 1433 may be performed by the communication interface 1414.
Computer system 1400 includes a bus 1406 or other communication mechanism for communicating instructions, messages and data, collectively, information, and one or more processors 1407 coupled with the bus 1406 for processing information. Computer system 1400 also includes a main memory 1408, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1406 for storing dynamic data and instructions to be executed by the processor(s) 1407. The main memory 1408 also may be used for storing temporary data, i.e., variables, or other intermediate information during execution of instructions by the processor(s) 1407. The computer system 1400 may further include a read only memory (ROM) 1409 or other static storage device coupled to the bus 1406 for storing static data and instructions for the processor(s) 1407. A storage device 1410, such as a magnetic disk or optical disk, may also be provided and coupled to the bus 1406 for storing data and instructions for the processor(s) 1407. A computer system 1400 may be coupled via the bus 1406 to a display device 1411, such as, but not limited to, a cathode ray tube (CRT), for displaying information to a user. An input device 1412, e.g., alphanumeric and other keys, is coupled to the bus 1406 for communicating information and command selections to the processor(s) 1407.
According to one embodiment of the invention, an individual computer system 1400 performs specific operations by their respective processor(s) 1407 executing one or more sequences of one or more instructions contained in the main memory 1408. Such instructions may be read into the main memory 1408 from another computer-usable medium, such as the ROM 1409 or the storage device 1410. Execution of the sequences of instructions contained in the main memory 1408 causes the processor(s) 1407 to perform the processes described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and/or software.
The term “computer-usable medium” or “computer-readable medium” as used herein, refers to any medium that provides information or is usable by the processor(s) 1407. Such a medium may take many forms, including, but not limited to, non-volatile, volatile and transmission media. Non-volatile media, i.e., media that can retain information in the absence of power, includes the ROM 1409, CD ROM, magnetic tape, and magnetic discs. Volatile media, i.e., media that can not retain information in the absence of power, includes the main memory 1408. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 1406. Transmission media can also take the form of carrier waves; i.e., electromagnetic waves that can be modulated, as in frequency, amplitude or phase, to transmit information signals. Additionally, transmission media can take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, the reader is to understand that the specific ordering and combination of process actions shown in the process flow diagrams described herein is merely illustrative, and the invention can be performed using different or additional process actions, or a different combination or ordering of process actions. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.
Number | Name | Date | Kind |
---|---|---|---|
5140685 | Sipple et al. | Aug 1992 | A |
5260697 | Barrett et al. | Nov 1993 | A |
5398183 | Elliott | Mar 1995 | A |
5408653 | Josten et al. | Apr 1995 | A |
5481712 | Silver et al. | Jan 1996 | A |
5504917 | Austin | Apr 1996 | A |
5577240 | Demers et al. | Nov 1996 | A |
5634134 | Kumai et al. | May 1997 | A |
5706510 | Burgoon | Jan 1998 | A |
5737601 | Jain et al. | Apr 1998 | A |
5761660 | Josten et al. | Jun 1998 | A |
5765159 | Srinivasan | Jun 1998 | A |
5781912 | Demers et al. | Jul 1998 | A |
5794229 | French et al. | Aug 1998 | A |
5806076 | Ngai et al. | Sep 1998 | A |
5870760 | Demers et al. | Feb 1999 | A |
5870761 | Demers et al. | Feb 1999 | A |
5940826 | Heideman et al. | Aug 1999 | A |
5991765 | Vethe | Nov 1999 | A |
6122640 | Pereira | Sep 2000 | A |
6178546 | McIntyre | Jan 2001 | B1 |
6195650 | Gaither et al. | Feb 2001 | B1 |
6349310 | Klein et al. | Feb 2002 | B1 |
6356889 | Lohman et al. | Mar 2002 | B1 |
6397227 | Klein et al. | May 2002 | B1 |
6411956 | Ng | Jun 2002 | B1 |
6434545 | MacLeod et al. | Aug 2002 | B1 |
6434568 | Bowman-Amuah | Aug 2002 | B1 |
6442748 | Bowman-Amuah | Aug 2002 | B1 |
6460027 | Cochrane et al. | Oct 2002 | B1 |
6493701 | Ponnekanti | Dec 2002 | B2 |
6496850 | Bowman-Amuah | Dec 2002 | B1 |
6560606 | Young | May 2003 | B1 |
6615223 | Shih et al. | Sep 2003 | B1 |
6701345 | Carley et al. | Mar 2004 | B1 |
6714943 | Ganesh et al. | Mar 2004 | B1 |
6725453 | Lucas et al. | Apr 2004 | B1 |
6728719 | Ganesh et al. | Apr 2004 | B1 |
6741997 | Liu et al. | May 2004 | B1 |
6744449 | MacLeod et al. | Jun 2004 | B2 |
6804672 | Klein et al. | Oct 2004 | B1 |
6856989 | Zhou et al. | Feb 2005 | B1 |
6910109 | Holman et al. | Jun 2005 | B2 |
6950833 | Costello et al. | Sep 2005 | B2 |
6961931 | Fischer | Nov 2005 | B2 |
6966058 | Earl et al. | Nov 2005 | B2 |
6996588 | Azagury et al. | Feb 2006 | B2 |
7080062 | Leung et al. | Jul 2006 | B1 |
7130897 | Dervin et al. | Oct 2006 | B2 |
7139749 | Bossman et al. | Nov 2006 | B2 |
7162477 | Mukherjee | Jan 2007 | B1 |
7505971 | Sander et al. | Mar 2009 | B2 |
20020198867 | Lohman et al. | Dec 2002 | A1 |
20030093408 | Brown et al. | May 2003 | A1 |
20030130985 | Driesen et al. | Jul 2003 | A1 |
20030135478 | Marshall et al. | Jul 2003 | A1 |
20030177137 | MacLeod et al. | Sep 2003 | A1 |
20030182276 | Bossman et al. | Sep 2003 | A1 |
20030229639 | Carlson et al. | Dec 2003 | A1 |
20040002990 | Sander et al. | Jan 2004 | A1 |
20040003004 | Chaudhuri et al. | Jan 2004 | A1 |
20040019587 | Fuh et al. | Jan 2004 | A1 |
20050097091 | Ramacher et al. | May 2005 | A1 |
20050119999 | Zait et al. | Jun 2005 | A1 |
20050120000 | Ziauddin et al. | Jun 2005 | A1 |
20050120001 | Yagoub et al. | Jun 2005 | A1 |
20050125393 | Yagoub et al. | Jun 2005 | A1 |
20050125398 | Das et al. | Jun 2005 | A1 |
20050125427 | Dageville et al. | Jun 2005 | A1 |
20050125452 | Ziauddin et al. | Jun 2005 | A1 |
20050138015 | Dageville et al. | Jun 2005 | A1 |
20050160419 | Alam et al. | Jul 2005 | A1 |
20050177557 | Ziauddin et al. | Aug 2005 | A1 |
20050187917 | Lawande et al. | Aug 2005 | A1 |
20050210461 | Srivastava et al. | Sep 2005 | A1 |
20050251523 | Rajamani et al. | Nov 2005 | A1 |
20060004828 | Rajamani et al. | Jan 2006 | A1 |
Number | Date | Country | |
---|---|---|---|
20050210461 A1 | Sep 2005 | US |