Distributed management of a storage system

Abstract

Systems, apparatuses, methods, and computer-readable storage mediums for performing lease-based fencing using a time-limited lease window. During the time-limited lease window, writes to a shared storage medium are permitted, while writes are denied for expired leases. When a successful heartbeat is generated for a primary storage controller, the lease window is extended for the primary storage controller from the time of a previous heartbeat. Accordingly, a prolonged stall between successive heartbeats by the primary storage controller will result in the newly extended lease being expired at the time it is granted. This scheme prevents a split brain scenario from occurring when a secondary storage controller takes over as the new primary storage controller in response to detecting the stall.

Description

BACKGROUND

Technical Field

Embodiments described herein relate to fencing and, more particularly, to techniques for implementing reliable lease-based fencing mechanisms for shared storage devices.

Description of the Related Art

Persistent reservations provide a mechanism for both heartbeating between several cluster members and performing access control to shared storage medium(s). However, existing approaches to fencing are unreliable due to shortcomings in common implementations. For example, a storage system may include a primary storage controller and a secondary storage controller coupled to a plurality of storage devices. When the primary storage controller experiences a delay or stall, the secondary storage controller may take over as the new primary storage controller. However, the old primary storage controller may return from the stall and attempt to perform an unauthorized access to the shared storage medium(s), potentially causing data corruption or other unintended behavior. To prevent this scenario and other similar scenarios, more reliable fencing mechanisms are needed.

SUMMARY OF EMBODIMENTS

Various embodiments of systems, apparatuses, methods, and computer readable storage mediums for implementing lease-based fencing are contemplated.

In one embodiment, multiple storage controllers may be coupled to one or more shared storage mediums for shared access and lease-based fencing may be utilized to control access to the shared stored mediums. Lease-based fencing may allow a primary storage controller to utilize a time-limited lease window during which accesses to the shared storage mediums are permitted, while accesses may be prohibited for expired leases. In one embodiment, write operations may be allowed for the primary controller with a current lease window and write operations may be prevented for one or more secondary controllers. In this embodiment, read operations may be allowed for all controllers.

In one embodiment, the primary storage controller may be configured to generate heartbeats at regularly spaced intervals (or heartbeat intervals). When the primary storage controller generates a successful heartbeat, the lease window may be extended, with the extension being calculated from a prior heartbeat. In one embodiment, the lease extension may be calculated from the previous (or most recent) heartbeat before the current heartbeat. In another embodiment, the lease extension may be calculated from a prior heartbeat performed two or more heartbeats before the current heartbeat. In a further embodiment, the lease extension may be calculated from a previous point in time when a prior lease extension was granted. In other embodiments, the lease extension may be calculated from a different point in time prior to the current heartbeat.

In one embodiment, if the secondary storage controller does not detect a heartbeat from the primary storage controller for a certain period of time (or takeover window), the secondary storage controller may take over as the new primary storage controller. If the old primary storage controller comes back up and generates a new heartbeat, it may be granted a lease extension which is calculated from a prior heartbeat. Accordingly, the lease extension will already be expired by the time it is granted if the takeover window is larger than the lease window. Therefore, the old primary storage controller will not be able to perform an unauthorized access to the shared storage medium(s) after the secondary storage controller has taken over as the new primary storage controller.

These and other embodiments will become apparent upon consideration of the following description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a generalized block diagram illustrating one embodiment of a storage system.

FIG. 2 is a generalized block diagram of one embodiment of two computing devices coupled to a shared resource.

FIG. 3 illustrates one embodiment of a timing diagram of a primary controller generating heartbeats and the corresponding lease extensions.

FIG. 4 illustrates one embodiment of a table storing entries corresponding to successful heartbeats.

FIG. 5 illustrates one embodiment of a timing diagram of a stalled primary controller.

FIG. 6 illustrates one embodiment of two tables storing entries corresponding to successful heartbeats.

FIG. 7 is a generalized flow diagram illustrating one embodiment of a method for performing operations of a primary storage controller.

FIG. 8 is a generalized flow diagram illustrating one embodiment of a method for allowing for state-changing operations to occur at any time without suffering a data loss due to a failure of the fencing and lease mechanisms.

While the invention is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, one having ordinary skill in the art should recognize that the invention might be practiced without these specific details. In some instances, well-known circuits, structures, signals, computer program instruction, and techniques have not been shown in detail to avoid obscuring the present invention. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

This specification includes references to “one embodiment”. The appearance of the phrase “in one embodiment” in different contexts does not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure. Furthermore, as used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean including, but not limited to.

Terminology. The following paragraphs provide definitions and/or context for terms found in this disclosure (including the appended claims):

“Comprising.” This term is open-ended. As used in the appended claims, this term does not foreclose additional structure or steps. Consider a claim that recites: “A system comprising a storage device . . . ” Such a claim does not foreclose the system from including additional components (e.g., a network interface, one or more processors, a storage controller).

“Configured To.” Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs the task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112, paragraph (f), for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configured to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.

“Based On.” As used herein, this term is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While B may be a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.

Referring now to FIG. 1, a generalized block diagram of one embodiment of a storage system 100 is shown. Storage system 100 may include storage controllers 105 and 110 and storage device groups 130 and 140. In other embodiments, storage system 100 may include other numbers of storage controllers. Storage device groups 130 and 140 are representative of any number of storage device groups (or data storage arrays) which may be included in storage system 100. As shown, storage device group 130 includes storage devices 135A-N, which are representative of any number and type of storage devices (e.g., solid-state drives (SSDs)). For example, in various embodiments, storage devices 135A-N may utilize a solid-state storage medium such as flash memory, static random access memory (SRAM), dynamic RAM (DRAM), etc. In other embodiments, storage devices 135A-N may be hard disk drives. In further embodiments, storage devices 135A-N may utilize both solid-state storage and hard disk drive storage. Storage device group 140 may also include a plurality of storage devices which are not shown to avoid obscuring the figure.

Each of storage controllers 105 and 110 may include software and/or hardware configured to provide access to storage devices 135A-N. Storage controllers 105 and 110 may be coupled directly to client computer system 125, and storage controllers 105 and 110 may be coupled remotely over network 120 to client computer system 115. Clients 115 and 125 are representative of any number of clients which may utilize storage controllers 105 and 110 for storing and accessing data in system 100. It is noted that some systems may include only a single client, connected directly or remotely to storage controllers 105 and 110. Although storage controllers 105 and 110 are shown as being separate from storage device groups 130 and 140, in some embodiments, portions or the entirety of storage controllers 105 and 110 may be located within one or each of storage device groups 130 and 140.

Storage controllers 105 and 110 may be connected to each other and to clients 115 and 125 using any suitable connection (e.g., local area network (LAN), storage area network (SAN)). In one embodiment, the connection between storage controllers 105 and 110 may be used for the delivery of heartbeat signals. Storage controllers 105 and 110 may also be connected to storage device groups 130 and 140 using any of a variety of connections. In one embodiment, the interfaces between controllers 105 and 110 and storage devices of storage device groups 130 and 140 may be custom designed interfaces. Alternatively, in other embodiments, these interfaces may utilize a standard communication protocol. For example, the interfaces between between controllers 105 and 110 and storage devices of storage device groups 130 and 140 may utilize a Serial Advanced Technology Attachment (“SATA”) bus protocol, a Small Computer System Interface (“SCSI”) bus protocol, a Serial Attached SCSI (“SAS”) bus protocol, a Peripheral Component Interconnect Express (PCIe) bus protocol, and/or any of various other communication protocols.

In a typical configuration, one of storage controllers 105 and 110 may be the primary storage controller and the other of storage controllers 105 and 110 may be the secondary storage controller. In systems with three or more controllers, one controller may be the primary controller and the other controllers may be secondary controllers. The various connections to and from storage controllers 105 and 110 may be configured to allow a high availability configuration and to allow either of storage controllers 105 and 110 to serve as the primary storage controller. In this configuration, the primary storage controller may have read and write access to storage device groups 130 and 140 while the secondary storage controller may have only read access to storage device groups 130 and 140. In other embodiments, other configurations may be utilized. For example, one storage controller may have access to a first portion of storage device groups 130 and 140 and the other storage controller may have access to a second portion of storage device groups 130 and 140, and access may be shared to a third portion of storage device groups 130 and 140. Additional configurations are possible and are contemplated.

Storage controllers 105 and 110 may include or be coupled to a base operating system (OS), a volume manager, and additional control logic for implementing the various techniques disclosed herein. Storage controllers 105 and 110 may also include and/or execute on any number of processors and may include and/or execute on a single computing device or be spread across multiple computing devices, depending on the embodiment. The computing device(s) may be servers, workstations, or other types of computing devices. In some embodiments, storage controllers 105 and 110 may include or execute on one or more file servers and/or block servers. Storage controllers 105 and 110 may use any of various techniques for replicating data across devices 135A-N to prevent loss of data due to the failure of a device or the failure of storage locations within a device.

Storage controllers 105 and 110 may include software and/or control logic for implementing a reliable lease-based fencing mechanism for shared storage devices 135A-N. In one embodiment, one of the storage controllers 105 and 110 may be designated as the primary controller and the other may be designated as the secondary controller. The primary storage controller may be configured to extend lease windows after establishing ownership of the shared storage devices 135A-N and may also be configured to enforce access failures on expired leases. In one embodiment, a valid lease may permit a storage controller to perform a write operation to a shared storage device on behalf of a client without having to check for permission from another storage controller that has access to the storage device. For example, in various embodiments the lease defines a time period during which the primary storage controller can perform write operations without further reference to any other storage controllers.

In one embodiment, enforcement of the lease may occur in the storage controller operating system kernel, at the lowest layer of the storage driver stack. However, it is contemplated that such enforcement may occur elsewhere within a given system. In one embodiment, the storage driver stack may be a SCSI stack. In one embodiment, enforcement may include performing command descriptor block (CDB) parsing to split SCSI commands into non-state-changing (or harmless) commands (e.g., reads, TEST_UNIT_READY, READ_CAPACITY, PERSISTENT_RESERVE_IN, MAINTENCE_IN) and state-changing (or dangerous) commands (e.g., writes, FORMAT_UNIT, ERASE, WRITE_SAME, UNMAP). The non-state-changing commands may be harmless no matter which or how many controllers perform them while the state-changing commands may become harmful when performed concurrently by multiple controllers. In some cases, there may be commands that are considered non-state-changing for a first system design which may be considered state-changing for a second system design. These commands may include SYNCHRONIZE_CACHE, SECURITY_PROTOCOL_OUT (SECURITY_UNLOCK), ENABLE_ADVANCED_POWER_MANAGEMENT, and SECURITY_DISABLE_PASSWORD. In some embodiments, non-state-changing commands may always be allowed while state-changing commands may only be permitted within the lease window.

Queueing times may cause a delay between issuing a command from userspace and the command reaching the storage device. Accordingly, queueing inside the kernel may be minimized in order to minimize these delays. However, since there is still queueing possible inside the storage devices, the lease window may be sized so that it is comfortably smaller than the takeover window, wherein the takeover window is defined as the amount of time without detecting a heartbeat that the secondary controller waits before taking over as the new primary controller.

It is noted that in alternative embodiments, the number and type of client computers, storage controllers, networks, storage device groups, and data storage devices is not limited to those shown in FIG. 1. At various times one or more clients may operate offline. In addition, during operation, individual client computer connection types may change as users connect, disconnect, and reconnect to system 100. Further, the systems and methods described herein may be applied to directly attached storage systems or network attached storage systems and may include a host operating system configured to perform one or more aspects of the described methods. Numerous such alternatives are possible and are contemplated.

Network 120 may utilize a variety of techniques including wireless connection, direct LAN connections, wide area network (WAN) connections such as the Internet, a router, storage area network, Ethernet, and others. Network 120 may comprise one or more LANs that may also be wireless. Network 120 may further include remote direct memory access (RDMA) hardware and/or software, transmission control protocol/internet protocol (TCP/IP) hardware and/or software, router, repeaters, switches, grids, and/or others. Protocols such as Fibre Channel, Fibre Channel over Ethernet (FCoE), iSCSI, and so forth may be used in network 120. The network 120 may interface with a set of communications protocols used for the Internet such as the Transmission Control Protocol (TCP) and the Internet Protocol (IP), or TCP/IP.

Client computer systems 115 and 125 are representative of any number of stationary or mobile computers such as desktop personal computers (PCs), servers, server farms, workstations, laptops, handheld computers, servers, personal digital assistants (PDAs), smart phones, and so forth. Generally speaking, client computer systems 115 and 125 include one or more processors comprising one or more processor cores. Each processor core includes circuitry for executing instructions according to a predefined general-purpose instruction set. For example, the x86 instruction set architecture may be selected. Alternatively, the ARM®, Alpha®, PowerPC®, SPARC®, or any other general-purpose instruction set architecture may be selected. The processor cores may access cache memory subsystems for data and computer program instructions. The cache subsystems may be coupled to a memory hierarchy comprising random access memory (RAM) and a storage device.

Turning now to FIG. 2, a block diagram of one embodiment two computing devices 205 and 210 coupled to a shared resource 215 is shown. It is noted that the two computing devices 205 and 210 are representative of any number and type of computing devices which may be coupled to shared resource 215. In addition, in some embodiments, computing devices 205 and 210 may be coupled to a plurality of shared resources, although this is not shown in FIG. 2. Each of computing devices 205 and 210 may be any type of computing device or system, such as a server, computer, smartphone, tablet, storage controller, virtual machine, or other entity which is capable of resource ownership. For example, in one embodiment, computing devices 205 and 210 may be storage controllers and resource 215 may be one or more storage devices, a storage shelf, or a portion of a storage device. Resource 215 may be any type of entity which permits shared access, such as a file, a record, a memory device, set of memory devices, a portion of a memory device, or other type of entity, depending on the embodiment.

In one embodiment, computing device 205 may operate as a primary device with read and write access to resource 215 while computing device 210 may operate as a secondary device with only read access to resource 215. In various embodiments, the device designated as the primary device (computing device 205 in this example,) may be configured to generate a heartbeat periodically (e.g., every N seconds, wherein N may be a fraction of a second). Depending on the embodiment, a heartbeat may be a signal or pulse generated by the computing device 205 and conveyed to computing device 210 and/or shared resource 215, a region of memory in resource 215 that is updated by computing device 205, or another suitable mechanism for indicating the health or status of computing device 205. In one embodiment, computing device 205 may generate a heartbeat every 100 milliseconds (ms) (or 0.1 seconds). In one embodiment, computing device 205 may issue itself a new lease window of M seconds every time it has generated a successful heartbeat, wherein M may be a fraction of a second, and wherein the lease window is extended from the time of a prior heartbeat rather than the current heartbeat. Alternatively, in another embodiment, the lease extension may be calculated from the time of a prior lease extension. In a further embodiment, the lease extension may be calculated from any point in time prior to the current heartbeat. For example, in this further embodiment, computing device 205 may take a timestamp and then perform the necessary verifications for confirming ownership of resource 215. After performing the necessary verifications, computing device 205 may extend the lease.

In one embodiment, computing device 210 may detect when heartbeats are generated by computing device 205, and if computing device 210 does not detect a heartbeat for a certain period of time (i.e., the takeover window), then computing device 210 may take over as the primary computing device 210. The takeover window may be programmable and may vary depending on the embodiment. After taking over as the primary device, if computing device 205 were to perform an unauthorized access to resource 215, such an unauthorized access could result in data corruption or other unintended outcomes.

Therefore, to prevent both computing devices from attempting to simultaneously control resource 215, a new lease extension may be calculated from the time of a prior lease extension. Depending on the embodiment, the lease extension may be calculated from the time of the heartbeat immediately prior to the current heartbeat, two heartbeats ago, three heartbeats ago, four heartbeats ago, or some other point in time in the past. By utilizing this scheme, at least a portion of the new lease window is for time that has already expired.

In the case where computing device 205 experiences a stall longer than the lease window between generating the heartbeat and extending the lease, the lease will already be expired when the lease gets extended for computing device 205 if the lease is extended from the previous heartbeat. For example, in one embodiment, if the lease window is 500 ms, and if computing device 205 experiences a stall for 1 second between heartbeats at times of 0 seconds and 1 second, then the new lease window granted for the heartbeat generated at time 1 second will be calculated from the previous heartbeat, which is 0 seconds. Accordingly, the lease will expire at 500 ms, which is in the past, and computing device 205 will have an expired lease at time 1 second preventing it from making an access to resource 215. This scheme allows computing device 205 to make one wrong decision after a stall and not cause a catastrophic failure. Therefore, when computing device 210 takes over as the new primary computing device at time 1 second, computing device 205 will be unable to access resource 215 and perform any actions which might corrupt or otherwise cause unintended behavior.

Referring now to FIG. 3, a timing diagram of one embodiment of a primary controller generating heartbeats and the corresponding lease extensions are shown. A healthy primary controller may generate heartbeats at fixed intervals in time, and the length of the fixed interval may be programmable and may vary according to the embodiment. As shown in FIG. 3, the interval for generating heartbeats is 100 ms, but this is used for illustrative purposes only. The primary storage controller may include a heartbeat interval counter for generating heartbeats, and the counter may be implemented using hardware or software.

The top row of FIG. 3 shows the generation of heartbeats every 100 ms (or 0.1 seconds) for the primary controller. The second row of FIG. 3 shows the extension of the lease window based on the heartbeat generated at time 0.1 seconds. The lease extension is determined by calculating the start of a new lease window from a point in time in the past. In this case, the point in time in the past is selected as the previous heartbeat at time 0 seconds. In other scenarios, the point in time in the past may vary. In this example, the lease window is 500 ms, but it is to be understood that the length of the lease window may be programmable and may have other values in other embodiments. Accordingly, the new lease window will expire at time 0.5 seconds for the heartbeat generated at time 0.1 seconds, which is an effective lease window of 0.4 seconds from the current time.

The third row of FIG. 3 shows the lease extension granted for the heartbeat generated at time 0.2 seconds. For this heartbeat, the lease window is calculated from the previous heartbeat which was generated at time 0.1 seconds. Therefore, the lease window of 500 ms is extended from 0.1 seconds to 0.6 seconds. Similarly, the fourth row shows the lease extension granted for the heartbeat generated at time 0.3 seconds. For this heartbeat, the lease extension is calculated from the time of the previously granted lease extension (at time 0.2 seconds). Therefore, the primary controller's lease now extends to 0.7 seconds. This pattern of extending the lease may continue for each heartbeat generated by the healthy primary controller. Lease extensions are also shown in FIG. 3 for the heartbeats generated at times 0.4 seconds and 0.5 seconds.

Turning now to FIG. 4, one embodiment of a table 400 storing entries corresponding to successfully generated heartbeats is shown. Table 400 includes entries 405A-E for the successfully generated heartbeats shown in the timing diagram of FIG. 3. Table 400 is intended to represent the timing of lease windows that are issued for each successful heartbeat and to show how the lease windows are dependent on previously generated heartbeats. In one embodiment, table 400 may be maintained by the primary storage controller and stored in the one or more shared storage devices. One or more secondary storage controller may also have access to table 400 in some embodiments. Although not shown in table 400, each entry in table 400 may also include an identifier of the storage controller which generated the heartbeat.

Entry 405A corresponds to the successful heartbeat generated at 0.1 seconds by the primary storage controller as shown in FIG. 3. The time of the previous heartbeat is 0 seconds, and so the lease window of 0.5 seconds is calculated from the previous heartbeat and therefore extends to 0.5 seconds. Accordingly, the lease window granted at time 0.1 seconds begins at a previous point in time (0 seconds) and expires at 0.5 seconds, giving an effective period of lease of 0.4 seconds.

This pattern continues for entry 405B which corresponds to the successful heartbeat performed at 0.2 seconds. As shown in entry 405B, the new lease window is valid from 0.1 seconds to 0.6 seconds. Similarly, entry 405C corresponds to the successful heartbeat performed at 0.3 seconds, and the lease window for this heartbeat will be calculated from the previous heartbeat of entry 405B (at 0.2 seconds) until 0.7 seconds when the new lease will expire. This pattern continues for entries 405D-E which will have their leases extended until 0.8 seconds and 0.9 seconds, respectively.

It is noted that in other embodiments, the lease window could be calculated from the time corresponding to two or more heartbeats prior to the current heartbeat. In other words, instead of calculating the lease extension from the time of the previous entry of table 400, the lease extension could be calculated from the time of the entry which is two or more entries prior to the current entry of table 400. For example, for entry 405D corresponding to the successful heartbeat at 0.4 seconds, the lease window could be calculated from entry 405A performed at 0.1 seconds. Accordingly, for this example, the lease window would go from 0.1 seconds to 0.6 seconds, giving an effective lease life of 0.2 seconds from the current time of 0.4 seconds. In other embodiments, the lease window may be calculated from other points in time from the past which are correlated to previous heartbeats, previous lease extensions, or other events or timestamps from the past.

Turning now to FIG. 5, a timing diagram of one embodiment of a stalled primary controller is shown. It may be assumed for the purposes of this discussion that the primary controller was functioning properly up until time 0 seconds. It may also be assumed that when healthy, the primary controller is configured to generate heartbeats every 0.1 seconds. It may also be assumed that the lease window is 0.5 seconds and that the takeover window is 1 second. In other embodiments, these values may vary within the constraints that the takeover window is greater than the lease window which is greater than the heartbeat interval. The primary controller generates heartbeats at −0.1 seconds and 0 seconds, and for the heartbeat generated at 0 seconds, the lease window is calculated from the heartbeat generated at −0.1 seconds. Accordingly, this lease window expires at 0.5 seconds.

After the heartbeat at 0 seconds, it may be assumed that the primary controller stops functioning normally and stalls rather than generating new heartbeats. As shown, the primary controller does not generate a heartbeat from time 0 seconds until time 1.1 seconds. The secondary controller may detect heartbeats generated by the primary controller, and when the secondary controller detects no new heartbeats for a length of time equal to or greater than the takeover window, the secondary controller may take over as the new primary controller. It is noted that in some embodiments, the lease window may be the same size or larger than the takeover window. In these embodiments, the secondary controller can perform part of a takeover, claiming all devices, before the lease expires. However, the secondary controller cannot trust data written to the devices before the lease expires. Therefore, the secondary controller may wait until the lease expires to perform any state-changing operations.

Accordingly, at time 1 second, a full takeover window has elapsed without a heartbeat, and so the secondary controller may generate a new heartbeat. Since this is the first heartbeat generated by the secondary controller, the secondary controller may not be given a lease for this heartbeat. On the next heartbeat generated by the secondary controller (the new primary controller) at 1.1 seconds, a new lease window may be granted and calculated from the previous heartbeat at time 1 second. Therefore, this lease window will extend from 1 second to 1.5 seconds. The secondary controller (or new primary controller) may continue to generate new heartbeats every 0.1 seconds and new lease windows may be granted for each new heartbeat, with the lease window being calculated from a previous heartbeat.

In some cases, the original primary controller may come back up after the stall and attempt to assert itself as the current primary controller. This is shown with the original primary controller generating a heartbeat at 1.1 seconds. For this heartbeat, a lease window may be granted, but this lease window will begin at the previous heartbeat generated by the original primary controller at 0 seconds. Accordingly, this lease window will already be expired at the time it is granted. Therefore, the original primary controller will be prevented from performing an access to the shared storage device(s) and a split-brain scenario will be avoided. In response to detecting the new lease window being expired at the time it is issued, the original primary controller may perform a check to determine if another storage controller has taken over as the new primary controller. In one embodiment, the original primary controller may detect heartbeats generated by the new primary controller and in response, the original primary controller may transition to the role of secondary controller.

Turning now to FIG. 6, one embodiment of tables 600 and 610 storing entries corresponding to successfully generated heartbeats are shown. Table 600 corresponds to the original primary storage controller of the timing diagram of FIG. 5. The primary storage controller generated successful heartbeats at times −0.1 second and 0 seconds and then experienced a stall of over a second before generating another heartbeat at time 1.1 seconds. Entry 605A corresponds to the heartbeat at 0 seconds and entry 605B corresponds to the heartbeat at 1.1 seconds. As can be seen from entry 605B, the lease window is calculated from the heartbeat of entry 605A (at 0 seconds) and expires at 0.5 seconds. Therefore, this lease window is already expired at the time it is granted, preventing the original primary storage controller from performing any write operations to the shared stored device(s) at time 1.1 seconds or later.

As shown in the timing diagram of FIG. 5, the secondary controller may detect the absence of heartbeats for a length of time equal to or greater than the takeover window (or 1 second). Accordingly, when no heartbeats are generated by the original primary controller from 0 seconds to 1 second, the secondary controller may take over as the new primary controller. Table 610 shows entries corresponding to the heartbeats generated by the original secondary controller acting as the new primary controller. The new primary controller may generate a heartbeat at 1 second (corresponding to entry 615A), and a lease may not be granted at this point since this is the first heartbeat generated by the new primary controller. At the next heartbeat at 1.1 seconds for the new primary controller, a new lease may be granted and the new lease window may be calculated from the previous heartbeat (at 1 second) of entry 615A. Entry 615B shows the details of the new lease which extends from 1 second to 1.5 seconds.

Entry 615C corresponds to the heartbeat performed at time 1.2 seconds, and the lease window may be calculated from the time of the previous entry 615B (or 1.1 seconds). Accordingly, the new lease for entry 615C expires at 1.6 seconds. Similarly, entry 615D corresponds to the heartbeat performed at time 1.3 seconds, and the lease window may be calculated from the time of the previous entry 615C (or 1.2 seconds). Therefore, the new lease for entry 615D expires at 1.7 seconds. This pattern may continue for each newly generated heartbeat by the new primary controller. It is noted that tables 600 and 610 may be combined into a single table in some embodiments. To differentiate between heartbeats generated by different controllers, each entry may include an identifier (ID) of the controller which generated the heartbeat.

Turning now to FIG. 7, one embodiment of a method 700 for performing operations of a primary storage controller is shown. For purposes of discussion, the steps in this embodiment are shown in sequential order. It should be noted that in various embodiments of the method described below, one or more of the elements described may be performed concurrently, in a different order than shown, or may be omitted entirely. Other additional elements may also be performed as desired. Method 700 may be performed in any of the storage controllers described herein.

A heartbeat interval counter used for determining when to generate a heartbeat for a storage controller may be initialized (block 705). In one embodiment, the counter may be configured to count until 100 ms has elapsed. In other embodiments, the counter may be configured to count for other periods of time. Next, the storage controller may determine if the heartbeat interval counter has expired (conditional block 710).

If the heartbeat interval counter has expired (conditional block 710, “yes” leg), then the storage controller may generate a new heartbeat (block 715). After generating the heartbeat, the storage controller may extend the lease window from a previous point in time prior to the new heartbeat, wherein the lease window allows state-changing operations to be performed to the one or more shared storage device(s) (block 720). In one embodiment, the extension of the lease window may be calculated from a previously generated heartbeat rather than the most recently generated heartbeat. In other words, the lease window may be extended from the point in time when a previous lease extension was made. For example, if the current heartbeat is generated at 1 second, the storage controller may determine when the heartbeat prior to the current heartbeat was generated, and then extend the lease window from that point in time. Accordingly, if the previous heartbeat was performed at 0.9 seconds, then the lease window may be extended from 0.9 seconds. The length of the lease window (e.g., 500 ms) may vary according to the embodiment.

If the heartbeat interval counter has not expired (conditional block 710, “no” leg), then the storage controller may determine if there are any pending state-changing operations targeting the shared storage device(s) (conditional block 725). If there are any pending state-changing operations targeting the shared storage device(s) (conditional block 725, “yes” leg), then the storage controller may determine if it has a valid lease for writing to the shared storage device(s) (conditional block 730). If there are no pending state-changing operations targeting the shared storage device(s) (conditional block 725, “no” leg), then method 700 may return to conditional block 710 to determine if the heartbeat interval counter has expired.

If the storage controller currently has a valid lease for writing to the shared storage device(s) (conditional block 730, “yes” leg), then the storage controller may perform the pending state-changing operations to the shared storage device(s) (block 735). After block 735, method 700 may return to conditional block 710 to determine if the heartbeat interval counter has expired. If the storage controller does not have a valid lease for writing to the shared storage device(s) (conditional block 730, “no” leg), then the storage controller may prevent the pending state-changing operations from being written to the shared storage device(s) (block 740). It is noted that a different storage controller which has a valid lease may perform the pending state-changing operations to the shared storage device(s).

After block 740, the storage controller may perform a check to determine if another storage controller has taken over as the new primary storage controller (conditional block 745). In various embodiments, the storage controller may determine if another storage controller has taken over as the new primary storage controller by detecting heartbeats generated by another controller, checking the heartbeat status table(s) (e.g., tables 400, 600, 610) for heartbeats generated by other controllers, querying the other controllers, and/or utilizing various other techniques. If another storage controller has taken over as the new primary storage controller (conditional block 745, “yes” leg), then the storage controller may operate as a secondary storage controller by performing only read operations to the shared storage device(s) (block 750). After block 750, method 700 may end. If no other storage controller has taken over as the new primary storage controller (conditional block 745, “no” leg), then method 700 may return to conditional block 710 to determine if the heartbeat interval counter has expired.

It is noted that method 700 may be implemented with multiple processes in some embodiments. For example, in another embodiment, the storage controller may constantly monitor the heartbeat interval counter with one process and use one or more other processes to determine whether there are state-changing operations pending, determine if there is a valid lease for performing state-changing operations, perform state-changing operations, and/or detect one or more other conditions.

Turning now to FIG. 8, one embodiment of a method 800 for allowing for state-changing operations to occur at any time without suffering a data loss due to a failure of a lease or other mechanism is shown. For purposes of discussion, the steps in this embodiment are shown in sequential order. It should be noted that in various embodiments of the method described below, one or more of the elements described may be performed concurrently, in a different order than shown, or may be omitted entirely. Other additional elements may also be performed as desired. Method 800 may be performed in any of the storage controllers described herein.

A storage controller may detect a plurality of pending state-changing operations targeting one or more shared storage devices (block 805). In response to detecting the plurality of pending state-changing operations, the storage controller may determine if there is a valid lease for performing the pending state-changing operations to the shared storage device(s) (conditional block 810). If the lease is valid (conditional block 810, “yes” leg), then the storage controller may determine a maximum number of state-changing operations that can be performed (and allowed to be in progress) without risking loss or inadvertent corruption of data or storage devices in the event of a failure of the fencing mechanism in the presence of state-changing operations (block 815). Alternatively, the number of state-changing operations that can safely be performed without causing loss or corruption of data or storage devices in the event of a failure may have previously been determined. In other words, the storage controller may determine how many and which of the pending state-changing operations could be recovered from if the lease or fencing mechanism fails and these pending state-changing operations were performed after the lease expires. In one scenario, the lease mechanism may fail if there is an unexpectedly long delay between checking the validity of the lease and performing the state-changing operation(s). In some cases a valid lease may no longer exist when the state-changing operations are performed. If the lease is then determined to not be valid (conditional block 810, “no” leg), then the storage controller may attempt to obtain a new lease (block 820). After block 820, method 800 may return to block 810 to determine if there is a valid lease for performing the pending state-changing operations.

The maximum number of the pending state-changing operations that can be performed without causing irrevocable damage if the lease mechanism were to fail may be based on the protection scheme which is being utilized for the shared storage devices. For example, if the protection scheme can withstand a single device failing or having corrupt data, any number of state-changing operations can be performed to one device. For a protection scheme that protects only one device, if there are pending state-changing operations targeting multiple storage devices, then state-changing operations may only be performed on a single device before checking the validity of the lease. Alternatively, if the protection scheme (e.g., RAID 6) being utilized provides fault tolerance for up to two failed devices, then state-changing operations may only be performed on two devices before checking the validity of the lease. For other types of protection schemes, the maximum number of state-changing operations which can be performed after checking the validity of the lease may vary based on the number of errors or failed storage devices the particular protection scheme can withstand if the lease mechanism were to fail. The number of allowed state-changing operations may also be limited to allow for one or more devices to fail or be corrupted due to some issue unrelated, such as failures within the devices themselves. For example, with RAID-6, we may allow one outstanding state-changing operation so that one unrelated device failure can be tolerated as well as one corruption resulting from a state-changing operation that happened after lease expiry.

In another embodiment, rather than determining a maximum number of pending state-changing operations which can be performed based on the current protection scheme, the storage controller may allow for a fixed number (which may be programmable) of state-changing operations to be performed after checking the validity of the lease. For example, in one embodiment, only a single state-changing operation may be allowed to be performed to a single storage device in response to determining the lease is valid. After performing the single state-changing operation, the validity of the lease may be checked again, and the next state-changing operation may be performed if the lease is still valid, with this process being repeated for each remaining pending state-changing operation.

After block 815, the storage controller may perform the maximum number of permitted state-changing operations to the shared storage device(s) (block 825). Next, the storage controller may determine if there are still any pending state-changing operations (conditional block 830). If there are one or more pending state-changing operations (conditional block 830, “yes” leg), then method 800 may return to block 810 to determine if there is a valid lease for performing the pending state-changing operations. If there are no more pending state-changing operations (conditional block 830, “no” leg), then method 800 may end.

It is noted that the above-described embodiments may comprise software. In such an embodiment, the program instructions that implement the methods and/or mechanisms may be conveyed or stored on a non-transitory computer readable medium. Numerous types of media which are configured to store program instructions are available and include hard disks, floppy disks, CD-ROM, DVD, flash memory, Programmable ROMs (PROM), random access memory (RAM), and various other forms of volatile or non-volatile storage.

In various embodiments, one or more portions of the methods and mechanisms described herein may form part of a cloud-computing environment. In such embodiments, resources may be provided over the Internet as services according to one or more various models. Such models may include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). In IaaS, computer infrastructure is delivered as a service. In such a case, the computing equipment is generally owned and operated by the service provider. In the PaaS model, software tools and underlying equipment used by developers to develop software solutions may be provided as a service and hosted by the service provider. SaaS typically includes a service provider licensing software as a service on demand. The service provider may host the software, or may deploy the software to a customer for a given period of time. Numerous combinations of the above models are possible and are contemplated.

Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. A storage system comprising: a plurality of storage controllers configured to couple to one or more client computer systems via one or more data communications links; andone or more storage devices coupled to the plurality of storage controllers;wherein a first storage controller of the plurality of storage controllers is configured to: generate heartbeats on intervals of a first period of time;issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat;determine if a lease is valid;responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices;detect a plurality of pending state-changing operations targeting the first portion of the storage devices;determine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; andperform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; andwherein a second storage controller of the plurality of storage controllers is configured to: generate heartbeats on intervals of a second period of time;issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat;determine if a lease is valid; andresponsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; andwherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

2. The storage system as recited in claim 1, wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

3. The storage system as recited in claim 2, wherein the first storage controller is further configured to: recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; andperform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

4. The storage system as recited in claim 1, wherein the first storage controller is a primary storage controller for the first portion of the storage devices, and wherein a second storage controller of the plurality of storage controllers is configured to take over as a new primary storage controller for the first portion of the storage devices responsive to detecting that the first storage controller has not generated a heartbeat for a particular period of time.

5. The storage system as recited in claim 4, wherein responsive to taking over as the new primary storage controller for the first portion of the storage devices, the second storage controller is configured to: generate heartbeats on intervals of the first period of time;issue a new lease for the first portion of the storage devices responsive to generating a heartbeat, wherein a beginning of the new lease is calculated from a prior heartbeat and has a duration of a third period of time; anddetermine if a particular lease is valid prior to performing a state-changing operation on the first portion of the storage devices.

6. The storage system as recited in claim 1, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

7. The storage system as recited in claim 1, further comprising: sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; andsending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.

8. A plurality of storage controllers: wherein each storage controller is configured to couple to one or more client computer systems via one or more data communications links and each storage controller is coupled to a plurality of storage devices;wherein a first storage controller is configured to: generate heartbeats on intervals of a first period of time;issue, for a first portion of the storage devices, a lease responsive to generating a current heartbeat;determine if a lease is valid;responsive to determining that the lease is valid, perform a state-changing operation on the first portion of the storage devices;detect a plurality of pending state-changing operations targeting the first portion of the storage devices; anddetermine a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; andperform a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number; andwherein a second storage controller is configured to: generate heartbeats on intervals of a second period of time;issue, for a second portion of the storage devices, a lease responsive to generating a current heartbeat;determine if a lease is valid; andresponsive to determining that the lease is valid, perform a state-changing operation on the second portion of the storage devices; andwherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

9. The plurality of storage controllers as recited in claim 8, wherein the first storage controller is further configured to prevent the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

10. The plurality of storage controllers as recited in claim 9, wherein the first storage controller is further configured to: recheck if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; andperform a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

11. The plurality of storage controllers as recited in claim 8, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

12. The plurality of storage controllers as recited in claim 8, wherein: the first storage controller is configured to send, to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; andthe second storage controller is configured to send, to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.

13. A method comprising: generating, by a first storage controller, heartbeats on intervals of a first period of time;issuing, by the first storage controller for a first portion of the storage devices, a lease responsive to generating a current heartbeat;determining, by the first storage controller, if a lease is valid;responsive to determining that the lease is valid, perform by the first storage controller, a state-changing operation on the first portion of the storage devices;generating, by a second storage controller, heartbeats on intervals of a second period of time;issuing, by the second storage controller for a second portion of the storage devices, a lease responsive to generating a current heartbeat;determining, by the second storage controller, if a lease is valid;responsive to determining that the lease is valid, perform by the second storage controller, a state-changing operation on the second portion of the storage devices;detecting, by the first storage controller, a plurality of pending state-changing operations targeting the first portion of the storage devices;determining, by the first storage controller, a maximum number of state-changing operations that can be performed without causing data loss due to a failure of a fencing mechanism; andperforming, by the first storage controller, a first number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the first number is less than or equal to the maximum number;wherein the first storage controller and the second storage controller are configured to perform non-state-changing operations on both portions of the storage devices.

14. The method as recited in claim 13, further comprising preventing, by the first storage controller, the state-changing operation from being performed on the first portion of the storage devices without a valid lease, and wherein the second storage controller does not have a valid lease on the first portion of the storage devices and the second storage controller cannot perform any state-changing operations on the first portion of the storage devices.

15. The method as recited in claim 14, further comprising: rechecking, by the first storage controller, if the particular lease is valid responsive to performing the first number of state-changing operations on the first portion of the storage devices; andperforming, by the first storage controller, a second number of state-changing operations on the first portion of the storage devices responsive to determining the particular lease is valid, wherein the second number is less than or equal to the maximum number.

16. The method as recited in claim 13, wherein the second storage controller is further configured to prevent state-changing operations from being performed on the second portion of the storage devices without a valid lease, and wherein the first storage controller does not have a valid lease on the second portion of the storage devices and the first storage controller cannot perform any state-changing operations on the second portion of the storage devices.

17. The method as recited in claim 13, further comprising: sending, from the first storage controller to the second storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller; andsending, from the second storage controller to the first storage controller, heartbeats via a data communications bus between the first storage controller and the second storage controller.