Method for communicating a software-generated pulse waveform between two servers in a network

Information

  • Patent Grant
  • 6523131
  • Patent Number
    6,523,131
  • Date Filed
    Friday, September 8, 2000
    24 years ago
  • Date Issued
    Tuesday, February 18, 2003
    21 years ago
Abstract
A method of monitoring a status condition of a first server with a second server in a server network, and also providing synchronization and messaging between the two servers, the method including: transmitting a software-generated pulse waveform from the first server to a device coupled to the first server, wherein the software-generated pulse waveform is comprises a first command corresponding to a logic level low and a second command corresponding to a logic level high; setting said device to a first state during logic level lows of said pulse waveform and to a second state during logic level highs of said pulse waveform; receiving the software-generated pulse waveform with the second server by determining when said device is in the first state and when it is in the second state; and determining when said device no longer changes from the first state to the second state. In order to provide synchronization and messaging, said pulse waveform is frequency modulated in order to communicate specified commands and/or reference points between the two servers.
Description




COPYRIGHT RIGHTS




A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.




BACKGROUND OF THE INVENTION




1. Field of the Invention




The invention relates to communications between two computer systems. More particularly, the invention relates to providing communications between two servers in a server network, for monitoring the operational status of the servers, synchronizing events or actions initiated by the servers, and providing messaging capability between the two servers.




2. Description of the Related Technology




As computer systems and networks become more complex, various systems for promoting fault tolerance in these networks have been developed. One method of preventing network down-time due to the failure or removal of a fileserver from a server network, is to implement “server mirroring.” Server mirroring as it is currently implemented requires a primary server, a primary storage device, a backup server, a backup storage device and a unified operating system linking the two servers and storage devices. The purpose of the backup server is to resume the operations of the primary server should it become inoperational. An example of a mirrored server product is provided by Software Fault Tolerance Level 3 (SFT III) provided by NOVELL INC., 1555 North Technology Way, Orem, Utah, as an add-on to its NetWare 7 4.x product. SFT III maintains servers in an identical state of data update. It separates hardware-related operating system (OS) functions on the mirrored servers so that a fault on one hardware platform does not affect the other.




The server OS is designed to work in tandem with two servers. One server is designated as a primary server, and the other is a secondary server. The primary server is the main point of update; the secondary server is in a constant state of readiness to take over. Both servers receive all updates through a special link called a mirrored server link (MSL), which is dedicated to this purpose. The servers also communicate over the local area network (LAN) that they share in common, so that one knows if the other has failed even if the MSL has failed. When a failure occurs, the second server automatically takes over without interrupting communications in any user-detectable way. Each server monitors the other server's NetWare 7 Core Protocol (NCP) acknowledgments over the LAN to see that all requests for that server are serviced and that OSs are constantly maintained in a mirrored state.




When the primary server fails, the secondary server detects the failure and immediately takes over as the primary server. The failure is detected in one or both of two ways: the MSL link generates an error condition when no activity is noticed, or the servers communicate over the LAN, each one monitoring the other's NCP acknowledgment. The primary server is simply the first server of the pair that is brought up. It then becomes the server used at all times and it processes all requests. When the primary server fails, the secondary server is immediately substituted as the primary server with identical configurations. The switch-over is handled entirely at the server end, and work continues without any perceivable interruption.




Although server mirroring increases security against down-time caused by a failed server, it does so at a considerable cost. This method of providing fault tolerance requires the additional expense and complexity of standby hardware that is not used unless there is a failure in the primary server.




Another method of providing fault tolerance in a server network which does not require additional redundant (mirrored) hardware is referred to as “clustering” the servers in the network. Under one type of clustering method, a replicated Network Directory Database (NDD) operates in conjunction with server resident processes, running on a cooperating set of servers called a cluster, to remap a network resource to an alternate server, in the event of a primary server failure. A remappable resource is called a clustered resource. The records/objects in the replicated database contain for each clustered network resource, a primary and a secondary server affiliation. Initially, all users access a network resource through the server identified in the replicated database as being the primary server for the network resource. When server resident processes detect a failure of that primary server, the replicated database is updated to reflect the failure of the primary server, and to change the affiliation of that network resource from its primary to its backup server.




This remapping occurs transparently to whichever user/client is accessing the network resource.




As a result of the remapping, all users access the clustered network resource through the server identified in the replicated database as the backup server for the resource. When the primary server returns to service, the replicated resident processes detect a return to service of the primary server, the replicated database is again updated to reflect the resumed operation of the primary server. As a result of these latter updates to the replicated database, all users once again access the network resource through the server identified in the replicated database as the primary server for the clustered network resource. This remapping of clustered network resource affiliations also occurs transparently to whichever user/client is accessing the network resource, and returns the resource to its original fault tolerant state. A further discussion of the operation and theory of clustered networks is provided in a U.S. provisional patent application, entitled, “Clustering Of Computer Systems Using Uniform Object Naming And Distributed Software For Locating Objects,” which is listed above under the heading “Priority Claim.”




The clustering method of remapping the affiliation of a network resource, reduces the amount of hardware, software and processing overhead required to provide fault tolerance, when compared with the mirrored server technique. However, in both of these methods and systems, a rather inefficient and costly method of monitoring the status of each server in the network is utilized. In order to detect that a primary server has failed, for example, these methods require both a primary server and a secondary server to communicate messages and commands across a LAN line and to process received messages and commands in accordance with a specified monitoring protocol.




One drawback of this method of providing communications between two or more servers within a server network is that it relies on a dedicated communications line, the LAN line, to communicate messages between the servers in the network. The LAN line is a valuable system resource which should be allocated only when necessary. Additionally, communicating across the LAN line is not totally reliable. If the bandwidth capacity of the LAN line is reached, or if the LAN line becomes physically damaged, it will not be able to handle communications from one server to another. Therefore, in order to provide a reliable method of monitoring and/or communicating between servers, a secondary method of communicating in the event that the LAN line becomes disabled is typically required. One such prior art secondary method includes a first server writing data, commands or information to an intermediate hard drive connected to a SCSI bus and a second server which reads the data, commands or information from the hard drive. Therefore, the hard drive serves as an intermediate depository for communicating between the SCSI adapters of two or more servers. One problem with this approach is that it creates a dependency on that device which is often a central point of failure. For example, if the hard drive “crashes,” the two servers will not be able to communicate with each other.




A typical prior art LAN handshake protocol between two servers includes the following steps: a first adapter of a first server will send a NetWare7 Core Protocol (NCP) packet to a second adapter card of a second server in order to check whether a second server is handling all its requests. The first adapter card must then wait for the second adapter card to receive the NCP signal, process it, and then send a response, which contains the intranetware address of the second adapter card. If the first adapter does not receive the intranetware address data in response to its NCP packet, the first adapter will wait for a specified amount of time after which the handshake protocol “times out” and ends, resulting in a failure to achieve a communications link with the first server.




This approach is time-consuming and requires much “overhead” in terms of processing time and logic circuitry to process and synchronize the series of commands and data transferred between the adapters of two servers trying to communicate with one another. Therefore, what is needed is a method and system for establishing communications between two or more servers within a server network such that the status of a server may be monitored, events and actions initiated by the servers may be synchronized with one another, and two or more servers may communicate with one another in a cost efficient and reliable manner. Additionally, such a method and system should reduce the amount of required “overhead”, in terms of processing time and system resources.




SUMMARY OF THE INVENTION




The invention addresses the above and other needs by providing a method and system of communicating between two servers of a server network so as to monitor the status of each server by the other, to time and/or synchronize the events and actions initiated by one server with respect to the other, and further to provide bi-directional messaging capability between the two servers.




In one embodiment of the invention, a method of monitoring an operational status of a first server with a second server, includes: successively transmitting first and second command signals to a device coupled to the first server, wherein the first command signal places the device in a first status condition and the second command signal places the device in a second status condition; and monitoring a status condition of the device with the second server, coupled to the device, wherein a change in the status condition of the device indicates that the first server is operational.




In another embodiment, a method of monitoring a status condition of a first server with a second server in a server network, includes: transmitting a software-generated pulse waveform from the first server to a device coupled to the first server, wherein the software-generated pulse waveform comprises a first command corresponding to a logic level low and a second command corresponding to a logic level high; setting the device to a first state during logic level lows of the pulse waveform and to a second state during logic level highs of the pulse waveform; receiving the software-generated pulse waveform with the second server by determining when the device is in the first state and when it is in the second state; and determining when the device no longer changes from the first state to the second state.




In a further embodiment, a method of monitoring a status condition of a first server by a second server, includes: transmitting SCSI Reserve and Release commands from the server to a SCSI device, coupled to the first server; and monitoring a released/reserved status of the SCSI device with the second server.




In yet a further embodiment, a method of assigning control over a network resource, includes: transmitting SCSI Reserve and Release commands from a first server to a SCSI device, coupled to the first server; monitoring a released/reserved status of the SCSI device with a second server; determining if the first server is operational; and if it is determined that the first server has failed, assigning control over the SCSI device to the second server.




In another embodiment, a method of synchronizing a first operation carried out by a first server with a second operation carried out by a second server, includes: ransmitting a software-generated pulse waveform, having a first frequency, from the first server to a device coupled to the first server; receiving the pulse waveform with the second server by monitoring a status condition of the device; transmitting from the first server a synchronization signal to the device by changing the frequency of the pulse waveform to a second frequency; detecting by the second server the synchronization signal by detecting a change in frequency of the pulse waveform; changing at the first server the frequency of the pulse waveform back to the first frequency; detecting by the second server a change in frequency from the second frequency back to the first frequency; and setting in both servers a reference point in time at a beginning of a first cycle of the pulse waveform after it has returned to the first frequency.




In yet another embodiment, a method of providing communications between a first server and a second server, includes: transmitting a first software-generated pulse waveform from the first server to a first device coupled to the first server, wherein the first pulse waveform changes a status condition of the first device between a first state and a second state; receiving the first software-generated pulse waveform with the second server by sampling the status condition of the first device; frequency modulating the first pulse waveform so as to encode a message into the pulse waveform; and reading the message with the second server by sampling the status condition of the first device at a predetermined first sampling rate.




In another embodiment, a method of providing communications between a first server and a second server, includes: executing a first pulse transmitter program in the first server so as to transmit a first software-generated pulse waveform from the first server to a first device coupled to the first server, wherein the pulse waveform changes a status condition of the first device between a first state and a second state; executing a first pulse receiver program in the second server so as to receive the first software-generated pulse waveform by sampling the status condition of the first device; frequency modulating the first pulse waveform so as to encode a first message into the first pulse waveform; reading the first message with the second server by sampling the status condition of the first device at a predetermined first sampling rate; executing a second pulse transmitter program in the second server for transmitting a second software-generated pulse waveform from the second server to a second device coupled to the second server, wherein the second pulse waveform changes a status condition of the second device between a third state and a fourth state; executing a second pulse receiver program in the first server for receiving the second software-generated pulse waveform with the first server by sampling the status condition of the second device; frequency modulating the second pulse waveform so as to encode a second message into the second pulse waveform; and reading the second message with the first server by sampling the status condition of the second device at a predetermined second sampling rate.











BRIEF DESCRIPTION OF THE DRAWINGS





FIG. 1

is a block diagram of a clustered server network having two file servers coupled to a SCSI device in accordance with one embodiment of the invention.





FIG. 2

illustrates a pulse waveform representing the reserved (logic high) and released (logic low) states of a SCSI device.





FIG. 3

is flowchart diagram illustrating the steps of one embodiment of a pulse transmitter software program for generating a software-generated pulse waveform in accordance with the invention.





FIG. 4

illustrates one method of sampling the software-generated pulse waveform by determining if the SCSI device is reserved or released at a predetermined sampling frequency.





FIG. 5

is a flowchart diagram illustrating the steps of one embodiment of a pulse receiver software program for receiving the software-generated pulse waveform in accordance with the invention.





FIG. 6

illustrates a pulse waveform being sampled at predetermined reference points, 180 degrees apart on the waveform, for purposes of monitoring the waveform for its presence in accordance with one embodiment of the invention.





FIG. 7

illustrates a pulse waveform being sampled at predetermined reference points, which are several cycles apart on the waveform, for purposes of monitoring the waveform for its presence in accordance with the invention.





FIG. 8

is a flowchart diagram illustrating one method of monitoring the pulse waveform in accordance with one embodiment of the invention.





FIG. 9

illustrates a modulated pulse waveform which may be utilized for the purpose of timing and/or synchronizing two servers with respect to one another in accordance with one embodiment of the invention.





FIG. 10

is a flowchart diagram illustrating one method of modulating the pulse waveform in order to provide a timing and synchronization signal for two servers in accordance with one embodiment of the invention.





FIG. 11

illustrates a modulated pulse waveform for providing messaging capability between two servers in accordance with one embodiment of the invention.





FIG. 12

illustrates a messaging protocol that may be utilized by two servers in accordance with one embodiment of the invention.





FIGS. 13A-13C

together form a flowchart diagram illustrating a messaging protocol between two servers in accordance with one embodiment of the invention.











DETAILED DESCRIPTION OF THE INVENTION




The invention is described in detail below with reference to the figures, wherein like elements are referenced with like numerals throughout.




The invention utilizes command signals sent from a SCSI adapter card to a SCSI device in order to reserve and release access time to that device by that server. Referring to

FIG. 1

, a block diagram of one embodiment of a clustered server network


100


is illustrated. The server network


100


includes a first file server computer


101


having a first SCSI host adapter card


103


coupled thereto. The first adapter card


103


is coupled to a SCSI device


105


for communicating commands and data to and from the SCSI device


105


by means of a first SCSI bus


107


. The server network


100


also includes a second file server computer


109


having a second SCSI host adapter card


111


contained therein. The second adapter card


111


is also connected to the SCSI device


105


by means of a second SCSI bus


113


. In another embodiment, each of the servers,


101


and


109


, may be connected to two or more common SCSI devices, such as SCSI device


105


, in order to provide fault tolerance and redundancy in the event that the SCSI device


105


becomes inoperational or otherwise damaged.




Typically, during normal operating conditions, only one server is allowed access and control of a single SCSI device at any one time. In order to arbitrate access and control over a SCSI device between multiple servers, a second SCSI host protocol is typically used in order to provide this function. In such a protocol, only one server is designated as a host server to the SCSI device and other servers may not access the device when the host server is accessing or desires to access the device. This protocol is accomplished by the SCSI commands of Reserve Unit, Release Unit, and Test Unit Ready, which are transmitted to the SCSI device by the servers connected to that device.




However, as described above, in a clustered server network, the assignment and control of host and backup status to each of the servers in the network is accomplished by means of a cluster data software program which maps and remaps the affiliation of a particular device to a particular server. Therefore, in a clustered network, the use of the SCSI Reserve and Release commands are not necessary to establish which server is the host server of a particular SCSI device. Therefore, the SCSI command protocol for establishing access rights to a SCSI device are no longer necessary and these command signals are free to be manipulated and utilized for other purposes.




Embodiments of the invention take advantage of these idle command signals and utilize the Reserve Unit, Release Unit and Test Unit Ready commands in order to communicate from one server to another. The Reserve Unit command and Release Unit command may serve as unique logic levels, while the Test Unit Ready command is used to read and monitor these “logic levels.” By manipulating the Reserve and Release commands a “software generated pulse waveform” may be created to communicate messages from one server to another.




In order to generate this software-generated pulse waveform, at least the host server is encoded with a “Pulse Transmitter” software program (hereinafter “pulse transmitter” or “pulse transmitter module”) which generates the Reserve and Release signals, or commands, and transmits them to a SCSI device. The processing time and circuitry overhead to create this pulse waveform is nominal. As used herein, the term “module” refers to any software program, subprogram, subroutine of a program, or software code capable of performing a specified function. Also, as used herein, the terms “command,”“signal” and “data” and any conjugations and combinations thereof, are used synonymously and interchangeably and refer to any information or value that may be transmitted, received or communicated between two electronic systems.




To further illustrate the concept of creating a software-generated pulse waveform, reference is made to FIG.


2


. In this figure, the Reserve command is represented as a logic level high and the Release command is represented as a logic level low. However, it should be kept in mind that this is only a convenient way of “labelling” these commands for the purpose of using them as a signalling tool. These commands in actuality are streams of data which are transmitted to a SCSI device for the purpose of reserving or releasing access to the device. As shown in

FIG. 2

, the time that the device is reserved is represented by a logic level high beginning at a rising edge


201


and ending at a falling edge


203


. The period of time between the rising and falling edges


201


and


203


, respectively, is referred to as the “Reservation time” (Rvt). The time that the device is released is represented by a logic level low beginning at the falling edge


203


and ending at a rising edge


205


. The period of time between the falling and rising edges


203


and


205


, respectively, is referred to as the “Release time” (Rlt). For the purpose of providing a signal that a host server


101


, is “alive” and operational, the pulse waveform may be uniform, that is Rvt=Rlt. However, it is not required that it be uniform. For example, Rvt may equal ½×Rlt. Furthermore, as described in further detail below, for the purpose of providing timing and synchronization signals, and messaging capability, this pulse waveform may be frequency modulated, for example, in order to provide messages, commands, timing reference points, etc.




In order to transmit this pulse waveform to the SCSI device, the pulse transmitter software code makes a call to a specified SCSI driver program contained within a hard drive that is currently loaded in memory and executed on the host server


101


. The SCSI driver then utilizes the SCSI adapter card


103


, otherwise known as the SCSI initiator or SCSI board, to send either a Reserve or Release command to the SCSI device.




Referring to

FIG. 3

, a flowchart diagram of one embodiment of the software code for transmitting a software pulse waveform to a SCSI device is illustrated. The process begins at start


300


and proceeds to step


301


in which the pulse transmitter sends a Reserve command to the device. In step


303


, the pulse transmitter will wait until a prespecified amount of reservation time (Rlt) has elapsed. In step


305


, the pulse transmitter determines whether to continue pulse generation. If in step


305


it is determined to discontinue pulse generation, the process moves to step


307


where it ends. On the otherhand, if in step


305


it is determined that the pulse transmitter is to continue pulse generation, in step


309


, it will send a Release command to the device. In step


311


, the pulse transmitter will then wait until a prespecified period of release time (Rlt) has elapsed. In step


313


, the pulse transmitter once again determines whether to continue pulse generation. If the answer is no, the process goes to step


307


where it ends. If the answer is yes, the process loops back to step


301


so that the above steps may be repeated.




In order to communicate between the host server


101


and the secondary, or backup, server


109


, a “Pulse Receiver” program (hereinafter “pulse receiver” or “pulse receiver module”) within the backup server


109


may listen to the pulse generated by the pulse transmitter of the host server. Similar to the pulse transmitter within the host server


101


, the pulse receiver is a software program stored within a memory of the backup server


109


and executed by the backup server


109


. To detect the state of the pulse, the pulse receiver may send a “Test Unit Ready” command to the device


105


(FIG.


1


). Upon receiving the Test command, the device


105


may indicate whether it is reserved, indicating a failed test, or released, indicating a successful test by transmitting back to the pulse receiver, response data which contains information pertaining to its reserved/released status. The rate at which the “Test Unit Ready” command is transmitted to the SCSI device


105


, otherwise known as the sample rate herein, is typically much faster than the rate at which the pulse waveform changes state. Therefore, the “Test Unit Ready” command in a sense “samples” the reservation time period (Rvt) and the release time period (Rlt) in order to ascertain the shape and frequency of the software pulse waveform.




Referring to

FIG. 4

, a graphical representation of the Test Unit Ready command sampling protocol is illustrated. The results of the Test Unit Ready command are indicated by either an “S” or a “F”, wherein S indicates a successful test when the device


105


is released by the host server


101


and F indicates a failed test when the device


105


is reserved by the host server


101


. As shown in

FIG. 4

, the released status is represented as a logic level low at elements


401


and


403


of the software pulse waveform


400


and the reserved status is represented as logic level highs at elements


405


and


407


of the software pulse waveform


400


. Above the pulse waveform


400


, the results of the “Test Unit Ready” command are indicated by S and F, wherein S (successful test) corresponds to the released status and F (failed test) corresponds to the reserved status. It should be noted that since a logic level high was arbitrarily selected as representing the reserved status and the logic level low was arbitrarily selected as representing the released status, these representations can be switched and still provide the signalling functionality as described above in accordance with the invention.





FIG. 5

illustrates a flowchart diagram of one embodiment of a pulse receiving process in accordance with the invention. The process starts at


500


and proceeds to step


501


wherein the secondary server


109


(

FIG. 1

) sends a Test Unit Ready command to the SCSI device


107


(

FIG. 1

) in response to which the device


107


will transmit data back to the secondary server


109


. In step


503


, the secondary server


109


determines whether the device


107


is ready to accept a SCSI command by processing the response data received by the device


107


. If the result of the Test Unit Ready command is a success, the process moves to step


505


where the secondary server


109


records that the status of the device


107


is released, reflecting a “S” in the software generated pulse waveform as shown in FIG.


4


. If the result of the Test Unit Ready command is a failure, the process moves to step


507


where the secondary server


109


records that the status of the device


107


is reserved, reflecting a “F” in the software generated pulse waveform as shown in FIG.


4


. In step


509


, the secondary server


109


waits until the next sample of the pulse waveform is to be taken. In step


511


, a determination is made as to whether to continue taking samples. If the answer is Yes, the process goes back to step


501


and the above process steps are repeated. If the answer is No, the process ends at step


513


.




The foregoing describes one embodiment of a process for transmitting a software-generated pulse waveform by a first server


101


and receiving the software-generated pulse waveform by a second server


109


, in accordance with the invention. Some useful applications of the software-generated pulse waveform are described below.




Pulse Waveform as “Heartbeat” of Primary Server




The pulse waveform can be monitored by a secondary server to determine if a primary server is present and operational. The primary server can send a pulse waveform to the secondary server as described above in order to tell the secondary server that it is “alive and well.” In this way, the pulse waveform serves as a kind of “heartbeat” of the first server which the second server listens for. If the pulse transmitter sends a constant pulse, the waveform can be determined and predicted. This does not mean that the reservation time Rvt must be equal to the release time Rlt, but rather all Rvt's are predictable and all Rlt's are predictable. A pulse with a smaller Rvt minimizes the amount of time a device is reserved, while still creating a heartbeat.




In order to determine the shape, or period, of the pulse waveform, the second server sends a Test Unit Ready command to a SCSI device in order to sample the waveform as described above. The second server can ascertain the cyclic period of the pulse waveform by noting the transition points (from S to F or F to S) of the waveform and recording the number of samples taken between the transition points after several successive cycles of the waveform. Once the waveform is known, there is no need to keep sampling the waveform at the previous sampling rate in order to monitor the presence of the waveform. Instead, samples can be taken out of phase from the transition points and less frequently based on the known period of the waveform in order to ascertain its presence. By taking samples less frequently, the processing time and circuitry overhead of this communication and monitoring method is significantly reduced. This concept is described in greater detail below with reference to

FIGS. 6 and 7

.




Referring to

FIG. 6

, a software-generated pulse waveform is shown. After the second server has determined the period of the waveform


600


, it does not need to sample the waveform at the sampling rate. Instead the second server samples the waveform at a first reference point


601


and at a second reference point


603


approximately


1801


out of phase from the first reference point


601


. Because the second server has previously recorded the shape and period of the waveform, it sends a Test Unit Ready command to the SCSI device at a time corresponding to the reference point


601


and expects to see a “S” (a successful test result indicating a Released status). Similarly, at reference point


603


, it expects to see a “F” (a failed test result indicating a Reserved status). It is appreciated that the reduced number of Test Unit Ready commands that must be transmitted to the SCSI device and the reduced number of results that must be subsequently processed significantly reduces processing time and other system resources that must be allocated to perform this function.




To further decrease the overhead of monitoring the pulse waveform, samples can be taken several cycles apart on the pulse waveform as shown in FIG.


7


.

FIG. 7

illustrates a pulse waveform


700


, wherein the second server samples the waveform


700


at a first reference point


701


where it expects to see a release state. After several cycles of the waveform, the second server samples the waveform


700


at a second reference point


703


where it expects to see a reserve state.




By monitoring the pulse waveform as described above, the operational status of the first server can be determined by the second server. The first server is considered operational if the expected results of the monitoring process are obtained. However, if the results indicate a constant released state, either the monitoring mechanism (the reference points) are out of synchronization or the first server is dead and the heartbeat has flatlined. Going out of synchronization is possible, but not common. Therefore, in one embodiment, the monitoring process of the invention checks for this possibility by “recalibrating” the pulse receiving process. This is done by repeating the sampling process as described above with respect to

FIGS. 4 and 5

. In this way, the presence of the pulse can be reverified. Once the shape and cyclic period of the pulse waveform generated by the first server has been redetermined, the second server can once again begin monitoring the waveform at the recalibrated reference points. However, if the pulse waveform is no longer present, it is determined that the first server is dead. In a clustered system, the second server can send a command to the Network Directory Database (NDD) to inform the NDD that it is assuming control over the resources previously handled by the first server.

FIG. 8

illustrates a flowchart diagram of one embodiment of the process of monitoring the “heartbeat” of the first server. The process starts at


800


and proceeds to step


801


, where the first server executes the pulse transmitter by sending Reserve and Release commands to the SCSI device. In step


803


, the second server executes the pulse receiver by sending Test Unit Ready commands to the SCSI device, until at least one full cycle of the pulse waveform has been sampled. In step


805


, the second server determines the reference points at which the pulse waveform is to be subsequently sampled in order to monitor its presence. In step


807


, the second server waits until the next reference point of the waveform is to be sampled. In step


809


, the second server samples the waveform at the reference point and determines whether the expect result was obtained. If the expected result is obtained, the process moves to step


811


wherein the second server determines whether to continue monitoring the first server. If the answer is Yes, the process moves back to step


807


and proceeds once again from this step. If the answer is No, the process ends at step


817


.




If in step


809


, the expected results are not obtained, the process moves to step


813


in which the pulse waveform, or “heartbeat” is recalibrated as described above. In step


815


, the second server determines whether the heartbeat has flatlined, i.e., whether the pulse waveform is still present. If it is determined that the heartbeat is still there, the process moves back to step


807


and once again proceeds from there. However, if in step


815


it is determined that the heartbeat has flatlined, the first server is deemed dead, and the process ends at step


817


.




Pulse Waveform as Clock




Another use of the pulse generator is to provide a clock which may be used to synchronize time and events carried out by two or more servers.

FIG. 9

illustrates one method of using the pulse waveform as a synchronizing mechanism. Based on the heartbeat design, a pulse waveform


900


is generated by a first server which acts as a time master. A second server acts as a time slave and runs the pulse receiver program as described above with respect to

FIG. 5

to sample the pulse waveform. Similar in concept to a radio station sounding a bell to synchronize time, the time master will disrupt its usual pulse waveform and send a synchronization signal. In

FIG. 9

, this synchronization signal is indicated by a change in frequency starting at element


901


of the pulse waveform


900


. The change in frequency may generate a pulse waveform at half the original frequency, for example. At element


903


, the slave server, expecting to see a logic level high but seeing a logic level low instead, recognizes the change in frequency and restarts sampling of the pulse waveform. At element


905


of the waveform


900


, the slave server knows the frequency of the synchronization signal. At element


907


, the master server returns to normal frequency. The slave recognizes the change in frequency at element


909


when it expects to see a low level but instead sees a high level. The slave then marks the beginning of that cycle as “Time


0


”. The beginning of each cycle thereafter is successively numbered as T


1


, T


2


, etc. Therefore, both the master and slave servers now have a common reference point in time, T


0


, which can be used to synchronize processes and events occurring in the two servers.





FIG. 10

shows a flowchart of one embodiment of the process of synchronizing two servers in accordance with the invention. The process starts at step


1000


and proceeds to step


1010


where the first server (the time master) runs the pulse transmitter program. In step


1020


, the second server (the time slave) runs the pulse receiver program in order to sample and record the pulse waveform generated and transmitted by the time master. This sampling is performed at a sampling rate which is much faster than the frequency of the pulse waveform. In step


1030


, the master sends a synchronization signal by changing the frequency of the pulse waveform to one half its original frequency, for example. In step


1040


, the slave detects the synchronization signal. In step


1050


, the master returns to the normal frequency of the original pulse waveform. In step


1060


, the slave detects the change back to the original frequency and marks the beginning of the first cycle of the reinstated original frequency as “Time


0


”. In step


1070


, the second server determines if the perceived change in frequency of the pulse waveform is due to a flatline condition of the pulse waveform. If in step


1070


, it is determined that the pulse waveform has not flatlined, the process moves to step


1080


and the master and slave servers update their time counters and set time T


0


as a common reference point in time for synchronization purposes. The process then moves back to step


1070


and checks whether the pulse waveform has flatlined. Each time it is determined that the pulse waveform is still present, the master and slave servers increment, i.e., update, their timers by one. However, if in step


1070


, it is determined that the pulse waveform has flatlined, the process ends as step


1090


and synchronization has failed.




Pulse Waveform as Messaging Device




In the embodiment described above, the communication between the two servers is unidirectional (from master to slave) and the slave does not acknowledge synchronization with the master. A bidirectional clock can be implemented as an enhancement to the above timing mechanism in which the slave can acknowledge the detection of Time


0


so that the master receives validation of synchronization. In a bidirectional clock system, both servers run both the pulse transmitter and the pulse receiver programs. Also, a second SCSI device to which the second server is the host server must be provided in order to implement bidirectional communications between the first and second servers. With this arrangement, the second server “owns” the second SCSI device and transmits Reserve and Release commands to the second SCSI device and the first server samples and monitors the status of the second SCSI device as described above.




With a bidirectional communication protocol as described above, the invention may also be utilized to provide not only timing and synchronization between two servers but also messaging between the two servers. Since the pulse waveform can be bi-directional and there are no restrictions on the waveform, a messaging protocol can be established between the two servers via the SCSI bus using frequency modulation. A simple Request/Acknowledge protocol, with session control (the ability for both sides to initiate communication) may be used.




Referring to

FIG. 11

, one embodiment of a bi-directional pulse waveform message signal is illustrated. A message is incorporated into the pulse waveform by modulating the duration or pulse width of the Reserved state (indicated by a logic level high) of the waveform. As shown in

FIG. 11

, a “R” is indicated at element


1101


, a “T” is indicated at element


1103


and a “S” is indicated at element


1105


. A group of Rvt values, each separated by a constant Rlt make up a command or message. The message shown in

FIG. 11

is the “RTS” signal which is a request to send a command to the other server (begin communication session). This command signal as well as others is described in further detail below with respect to FIG.


12


. When no communication is occurring, a uniform Rvt=Rlt pulse is sent.




As mentioned above, to accommodate a bidirectional communication mechanism, both file servers must run the pulse transmitter and pulse receiver programs, and both servers must each “own” one device to which it can transmit Release and Reserve commands while the other server samples the status of that device at the sample rate.

FIG. 12

illustrates one example of a possible communication protocol between two file servers. A second server


109


first sends a Request to Send (RTS) command to the first server


101


The first server


101


then responds with a Ready to Receive (RTR) signal which establishes communications between the two servers. Next, the second server


109


sends a Request (REQ) signal to the first server


101


. The first server


101


then sends back either an Acknowledge (ACK) signal or Not Acknowledge (NAK) signal which indicates that the first server


101


can or cannot process the second server's request. After receiving a NAK signal, the second server


109


will terminate communications by sending a Relinquish (REL) to the first server


101


. If an acknowledge (ACK) signal is sent back, the first server


101


will process the request signal from the second server


109


and send a Result (RES) signal to the second server


109


after which the second server terminates communications by sending a Relinquish (REL) signal to the first server


101


.




The following is a summary of the messages/commands discussed above:




RTS: Request to send a command to the other file server (begin communication session).




RTR: Ready to receive-other server agrees to accept a command.




REQ: The command requesting certain actions/data on the part of the receiving server (defined by application).




ACK: The other server acknowledges the arrival and ability to service the command.




NAK: The other server does not support that command.




RES: The results of the command.




REL: The initiating server relinquishes control (end of session).




Referring to

FIGS. 13A-13C

a flowchart diagram of one embodiment of a bi-directional communication protocol in accordance with the invention is illustrated. The process starts at


1300


and proceeds to step


1301


wherein both servers start running the pulse transmitter program to generate a uniform pulse waveform wherein Rvt=Rlt. In step


1303


, both servers initiate the pulse receiver program to continuously sample the pulse waveform generated by the other server. As used herein the term “continuous sampling” means that the pulse receiver is sampling the pulse waveform at every “tick” or cycle of the sampling frequency, which is greater than the frequency of the pulse waveform, in order to determine the shape and frequency of the waveform, rather than sampling at only select reference points of the waveform for monitoring purposes.




In order to the simplify the discussion, the following description of the remaining portions of the flowchart is provided from the perspective of only one of the two servers, the first server. However, it is understood that the roles of the first and second servers are interchangeable in the following discussion. In step


1305


, the first server determines whether it has a message to send to the second server. If the first server does not have a message queued to send to the second server, the process moves to step


1307


wherein the first server determines whether a Request to Send (RTS) command has been sent by the second server. If the first server has not received a RTS signal from the second server, the process returns to step


1305


, wherein the first server continues to poll itself as to whether it has a message to send to the second server. If in step


1307


, it is determined that the second server has sent a RTS signal to the first server, the process moves to step


1309


as shown in FIG.


13


B. In step


1309


, the first server sends back a Ready to Receive (RTR) signal to the second server, agreeing to accept a command from the second server. Thereafter, in step


1311


, the first server receives the request (command) from the second server and decodes the request. In step


1313


, the first server determines whether it can support or accommodate the request from the second server. If it is determined that the first server can support the request, in step


1315


, the first server sends an acknowledgement (ACK) signal to the second server. In step


1317


, the first server passes the request up to an application software program running on the first server to process the request. In step


1319


, the first server sends the results of the application program to the second server. In step


1321


, the first server waits for a relinquish (REL) signal from the second server which terminates communications between the two servers and sends the process back to step


1305


of

FIG. 13A

at which point the first server once again resumes polling whether it has a message to send to the second server.




If in step


1313


, the first server determines that it does not support the request sent by the second server, in step


1323


, the first server sends a Not Acknowledge (NAK) signal to the second server which informs the second server that the first server does not support that command. The process then moves back to step


1305


wherein the first server continues to poll itself as to whether it has a message to send to the second server.




If in step


1305


, the first server determines that it has a message queued to be sent to the second server, the process moves to step


1325


of FIG.


13


C. In step


1325


, the first server sends a Request to Send (RTS) signal to the second server which requests the second server to accept a command from the first server. In step


1327


, the first server will wait for a Ready to Receive (RTR) from the second server. In step


1329


, the first server determines if a timeout period has expired. If the timeout period has expired before a RTR signal is received from the second server, the process moves to step


1331


in which the first server passes an error message to application software indicating that the second server is not responding. The process then moves back to step


1305


of FIG.


13


A.




If in step


1329


, the first server receives a RTR signal from the second server before the timeout period expires, the process moves to step


1333


in which the first server will send a request, or command, to the second server. In step


1335


, the first server determines whether an acknowledge (ACK) signal has been returned by the second server. If the second server does not send an ACK signal or, instead, sends back a Not Acknowledge (NAK) signal, the process moves back to step


1331


in which the first server sends an error signal to the application software indicating that the second server does not support the request. However, if in step


1335


, the first server receives the ACK signal from the second server, the process moves to step


1337


in which the first server receives the response (RES) signal(s) from the second server and passes the response to application software for processing. Thereafter, in step


1339


, the first server sends a relinquish (REL) signal to the second server thereby terminating communications with the second server. The process then moves back to step


1305


wherein the first server determines whether it has a message to send. If not, in step


1307


, the first server determines if the second server wants to send it a message. This process may be recursive and continue to execute until it is terminated by each of the servers.




As described above, the invention provides an efficient and reliable method and system for communicating between two servers in a server network. Because the communication method and system does not transmit data to be stored in an intermediate device, such as a hard disk drive, it does not depend on the presence or operational status of such a device. Rather, some embodiments of the invention take advantage of an existing protocol for establishing control of a SCSI device, namely, the Reserve and Release commands sent by a host server to a SCSI device. By sampling and monitoring the Reserve and Release status of the SCSI device by a second server which is also coupled to the device, the second device can monitor the presence and operational status of the host server, without directly communicating to the host server via an intermediate disk drive as is done in prior art systems. The Reserve and Release status of the SCSI device is used to generate a software-generated pulse waveform which serves as a sort of “heartbeat” of the host server and which can be monitored by the second server. Because the Reserve and Release commands are already implemented in existing SCSI device protocols, the method and system of the invention, once established, requires very little overhead, in terms of processing time and system resources, and is inexpensive to implement.




Because there are no limitations as to the shape and frequency of the software generated pulse waveform, the waveform may be frequency modulated in order to provide timing and synchronization signals to two servers. Additionally, by implementing each server as both a sender of Reserve and Release commands to a respective SCSI device, and a monitor of a the respective SCSI device, bidirectional communications can be established between the two servers. This method and system of communication between two servers is further advantageous in that it may be implemented with existing resources and command protocols between a server and a SCSI device and, additionally, it can be turned on and off as desired.




The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims, rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.



Claims
  • 1. A method of assigning control over a network resource, comprising:transmitting SCSI Reserve and Release commands from a first server to a SCSI device, coupled to the first server; monitoring a released/reserved status of the SCSI device with a second server; determining if the first server is operational; and if it is determined that the first server has failed, assigning control over the SCSI device to the second server.
  • 2. The method of claim 1 wherein the step of assigning control over the SCSI device to the second server comprises notifying a Network Directory Database that the first server has failed and that the second server is now the host server to the SCSI device.
  • 3. The method of claim 1 wherein a change in the released/reserved status of the SCSI device indicates that the first server remains operational and a constant reserved status of the SCSI device indicates that the first server has failed.
  • 4. A method of synchronizing a first operation carried out by a first server with a second operation carried out by a second server, comprising:transmitting a software-generated pulse waveform, having a first frequency, from the first server to a device coupled to the first server; receiving the pulse waveform with the second server by monitoring a status condition of the device; transmitting from the first server a synchronization signal to the device by changing the frequency of the pulse waveform to a second frequency; detecting by the second server the synchronization signal by detecting a change in frequency of the pulse waveform; changing at the first server the frequency of the pulse waveform back to the first frequency; detecting by the second server a change in frequency from the second frequency back to the first frequency; and setting in both servers a reference point in time at a beginning of a first cycle of the pulse waveform after it has returned to the first frequency.
  • 5. The method of claim 4 wherein:the pulse waveform is uniform when it is at the first frequency such that a first period of time corresponding to a logic level high of the pulse waveform is equal to a second period of time corresponding to a logic level low of the pulse waveform; the act of changing the frequency of the pulse waveform comprises changing at least one of the first and second periods of time; and the act of receiving the software-generated pulse waveform with the second server comprises sampling the status condition of the device at a predetermined sampling rate, wherein the logic level high of the pulse waveform sets a status condition of the device to a first state and the logic level low of the pulse waveform sets the status condition of the device to a second state.
  • 6. The method of claim 5 wherein:the logic level high of the pulse waveform is represented by a SCSI Reserve command which reserves access to the device exclusively to the first server; the logic level low of the pulse waveform is represented by a SCSI Release command which releases the device from exclusive access by the first server; and the act of sampling the status condition of the device comprises: repetitively transmitting a SCSI Test command from the second server to the device at the sampling rate; and receiving a response signal for each Test command which indicates the status condition of the device.
  • 7. A method of synchronizing a first operation carried out by a first server with a second operation carried out by a second server, comprising:transmitting a software-generated pulse waveform from the first server to a device coupled to the first server, wherein the pulse waveform is uniform such that a first period of time corresponding to a logic level high of the pulse waveform is equal to a second period of time corresponding to a logic level low of the pulse waveform, and wherein the logic level high of the pulse waveform sets a status condition of the device to a first state and the logic level low of the pulse waveform sets the status condition of the device to a second state; receiving the software-generated pulse waveform with the second server by sampling the status condition of the device; transmitting a synchronization signal from the first server to the device by frequency modulating the software-generated pulse waveform so as to vary at least one of the first and second periods; receiving the synchronization signal with the second server by detecting a change in frequency of the pulse waveform; and resuming transmission of the uniform pulse waveform to the device, wherein the second server detects the resumption in frequency and marks a beginning of a first cycle of the uniform pulse waveform, after the synchronization signal, as a reference point in time, and the first server also marks the beginning of the first cycle of the uniform pulse waveform as a reference point in time.
  • 8. A method of synchronizing a first operation performed by a first server with a second operation performed by a second server, comprising:executing a pulse transmitter program in the first server, said program successively transmitting SCSI Reserve and Release commands from the first server to a SCSI device so as to place the SCSI device in successive states of reserved and released status, wherein the states of the SCSI device serve as a basis for a software-generated pulse waveform; executing a pulse receiver program in the second server, said program sampling the software-generated pulse waveform at a predetermined sampling rate; determining when the pulse waveform has changed from a first frequency to a second frequency; and determining when the pulse waveform has changed from the second frequency back to the first frequency, wherein the first and second server each record a beginning of a first cycle of the pulse waveform after it has changed from the second frequency back to the first frequency as a common reference point in time.
  • 9. A method of providing communications between a first server and a second server, comprising:transmitting a first software-generated pulse waveform from the first server to a first device coupled to the first server, wherein the first pulse waveform changes a status condition of the first device between a first state and a second state; receiving the first software-generated pulse waveform with the second server by sampling the status condition of the first device; frequency modulating the first pulse waveform so as to encode a message into the pulse waveform; and reading the message with the second server by sampling the status condition of the first device at a predetermined first sampling rate.
  • 10. The method of claim 9 wherein when the first server is not sending a message to the second server, the first pulse waveform is uniform such that a first period of time corresponding to a logic level high of the first pulse waveform is equal to a second period of time corresponding to a logic level low of the first pulse waveform, and wherein the logic level high of the pulse waveform sets a status condition of the first device to a first state and the logic level low of the first pulse waveform sets the status condition of the first device to a second state.
  • 11. The method of claim 10 wherein:the act of receiving the first pulse waveform comprises sending a SCSI Test command to the first device at the first sampling rate and receiving a response from the first device as to its status condition; the logic level high of the first pulse waveform is represented by a SCSI Reserve command; the logic level low of the first pulse waveform is represented by a SCSI Release command; the first device is a first SCSI device; the first state is a reserved status condition; and the second state is a released status condition.
  • 12. The method of claim 11 further comprising:transmitting a second software-generated pulse waveform from the second server to a second device coupled to the second server, wherein the second pulse waveform changes a status condition of the second device between a third state and a fourth state; receiving the second software-generated pulse waveform with the first server by sampling the status condition of the second device; frequency modulating the second pulse waveform so as to encode a second message into the second pulse waveform; and reading the message with the first server by sampling the status condition of the second device at a predetermined second sampling rate.
  • 13. The method of claim 12 wherein when the second server is not sending a message to the first server, the second pulse waveform is uniform such that a third period of time corresponding to a logic level high of the second pulse waveform is equal to a fourth period of time corresponding to a logic level low of the second pulse waveform, and wherein the logic level high of the second pulse waveform sets the status condition of the second device to the third state and the logic level low of the second pulse waveform sets the status condition of the second device to the fourth state.
  • 14. The method of claim 13 wherein:the act of receiving the first pulse waveform comprises sending a SCSI Test command to the first device at the first sampling rate and receiving a response from the first device as to its status condition; the logic level high of the first pulse waveform is represented by a SCSI Reserve command; the logic level low of the first pulse waveform is represented by a SCSI Release command; the first device is a first SCSI device; the first state is a reserved status condition; the second state is a released status condition; the act of receiving the second pulse waveform comprises sending a SCSI Test command from the first server to the second device at the second sampling rate and receiving a response from the second SCSI device as to its status condition, the logic level high of the second pulse waveform is represented by a second SCSI Reserve command; the logic level low of the second pulse waveform is represented by a second SCSI Release command; the second device is a second SCSI device; the third state is a reserved status condition; and the fourth state is a released status condition.
  • 15. The method of claim 9 further comprising:transmitting a second software-generated pulse waveform from the second server to a second device coupled to the second server, wherein the second pulse waveform changes a status condition of the second device between a third state and a fourth state; receiving the second software-generated pulse waveform with the first server by sampling the status condition of the second device; frequency modulating the second pulse waveform so as to encode a second message into the second pulse waveform; and reading the message with the first server by sampling the status condition of the second device at a predetermined second sampling rate.
  • 16. A method of providing communications between a first server and a second server, comprising:executing a first pulse transmitter program in the first server so as to transmit a first software-generated pulse waveform from the first server to a first device coupled to the first server, wherein the pulse waveform changes a status condition of the first device between a first state and a second state; executing a first pulse receiver program in the second server so as to receive the first software-generated pulse waveform by sampling the status condition of the first device; frequency modulating the first pulse waveform so as to encode a first message into the first pulse waveform; reading the first message with the second server by sampling the status condition of the first device at a predetermined first sampling rate; executing a second pulse transmitter program in the second server for transmitting a second software-generated pulse waveform from the second server to a second device coupled to the second server, wherein the second pulse waveform changes a status condition of the second device between a third state and a fourth state; executing a second pulse receiver program in the first server for receiving the second software-generated pulse waveform with the first server by sampling the status condition of the second device; frequency modulating the second pulse waveform so as to encode a second message into the second pulse waveform; and reading the second message with the first server by sampling the status condition of the second device at a predetermined second sampling rate.
  • 17. The method of claim 16 wherein:when the first server is not sending the first message to the second server, the first pulse waveform is uniform such that a first period of time corresponding to a logic level high of the first pulse waveform is equal to a second period of time corresponding to a logic level low of the first pulse waveform, and wherein the logic level high of the pulse waveform sets a status condition of the first device to a first state and the logic level low of the first pulse waveform sets the status condition of the first device to a second state; and when the second server is not sending the second message to the first server, the second pulse waveform is uniform such that a third period of time corresponding to a logic level high of the second pulse waveform is equal to a fourth period of time corresponding to a logic level low of the second pulse waveform, and wherein the logic level high of the second pulse waveform sets the status condition of the second device to the third state and the logic level low of the second pulse waveform sets the status condition of the second device to the fourth state.
  • 18. The method of claim 17 wherein:the act of receiving the first pulse waveform comprises sending a SCSI Test command to the first device at the first sampling rate and receiving a response from the first device as to its status condition; the logic level high of the first pulse waveform is represented by a first SCSI Reserve command; the logic level low of the first pulse waveform is represented by a first SCSI Release command; the first device is a first SCSI device; the first state is a reserved status condition; the second state is a released status condition; the act of receiving the second pulse waveform comprises sending a SCSI Test command from the first server to the second device at the second sampling rate and receiving a response from the second SCSI device as to its status condition; the logic level high of the second pulse waveform is represented by a second SCSI Reserve command; the logic level low of the second pulse waveform is represented by a second SCSI Release command; the second device is a second SCSI device; the third state is a reserved status condition; and the fourth state is a released status condition.
RELATED APPLICATIONS

This application is a division of to U.S. patent application Ser. No. 08/942,221 entitled, “A System for Communicating a Software Generated Pulse Waveform Between Two Servers in a Network,” which was filed on Oct. 1, 1997, now U.S. Pat. No. 6,163,853 which claims benefit of provisional application Ser. No. 60/046,327 filed May 13, 1997. The following patent applications, commonly owned and filed Oct. 1, 1997, are hereby incorporated herein in their entirety by reference thereto:

US Referenced Citations (287)
Number Name Date Kind
4057847 Lowell et al. Nov 1977 A
4100597 Fleming et al. Jul 1978 A
4449182 Rubinson et al. May 1984 A
4672535 Katzman et al. Jun 1987 A
4692918 Elliott et al. Sep 1987 A
4695946 Andreasen et al. Sep 1987 A
4707803 Anthony, Jr. et al. Nov 1987 A
4769764 Levanon Sep 1988 A
4774502 Kimura Sep 1988 A
4821180 Gerety et al. Apr 1989 A
4835737 Herrig et al. May 1989 A
4894792 Mitchell et al. Jan 1990 A
4949245 Martin et al. Aug 1990 A
4999787 McNally et al. Mar 1991 A
5006961 Monico Apr 1991 A
5007431 Donehoo, III Apr 1991 A
5033048 Pierce et al. Jul 1991 A
5051720 Kittirutsunetorn Sep 1991 A
5073932 Yossifor et al. Dec 1991 A
5103391 Barrett Apr 1992 A
5118970 Olson et al. Jun 1992 A
5121500 Arlington et al. Jun 1992 A
5136708 Lapourtre et al. Aug 1992 A
5138619 Fasang et al. Aug 1992 A
5157663 Major et al. Oct 1992 A
5210855 Bartol May 1993 A
5245615 Treu Sep 1993 A
5247683 Holmes et al. Sep 1993 A
5253348 Scalise Oct 1993 A
5265098 Mattson et al. Nov 1993 A
5266838 Gerner Nov 1993 A
5269011 Yanai et al. Dec 1993 A
5272382 Heald et al. Dec 1993 A
5272584 Austruy et al. Dec 1993 A
5276863 Heider Jan 1994 A
5277615 Hastings et al. Jan 1994 A
5280621 Barnes et al. Jan 1994 A
5283905 Saadeh et al. Feb 1994 A
5307354 Cramer et al. Apr 1994 A
5311397 Harshberger et al. May 1994 A
5311451 Barrett May 1994 A
5317693 Cuenod et al. May 1994 A
5329625 Kannan et al. Jul 1994 A
5337413 Lui et al. Aug 1994 A
5351276 Doll, Jr. et al. Sep 1994 A
5367670 Ward et al. Nov 1994 A
5379184 Barraza et al. Jan 1995 A
5386567 Lien et al. Jan 1995 A
5388267 Chan et al. Feb 1995 A
5402431 Saadeh et al. Mar 1995 A
5404494 Garney Apr 1995 A
5423025 Goldman et al. Jun 1995 A
5430717 Fowler et al. Jul 1995 A
5430845 Rimmer et al. Jul 1995 A
5432715 Shigematsu et al. Jul 1995 A
5432946 Allard et al. Jul 1995 A
5438678 Smith Aug 1995 A
5440748 Sekine et al. Aug 1995 A
5448723 Rowett Sep 1995 A
5455933 Schieve et al. Oct 1995 A
5460441 Hastings et al. Oct 1995 A
5463766 Schieve et al. Oct 1995 A
5471617 Farrand et al. Nov 1995 A
5471634 Giorgio et al. Nov 1995 A
5473499 Weir Dec 1995 A
5483419 Kaczeus, Sr. et al. Jan 1996 A
5485550 Dalton Jan 1996 A
5487148 Komori et al. Jan 1996 A
5491791 Glowny et al. Feb 1996 A
5493574 McKinley Feb 1996 A
5493666 Fitch Feb 1996 A
5513314 Kandasamy et al. Apr 1996 A
5513339 Agrawal et al. Apr 1996 A
5517646 Piccirillo et al. May 1996 A
5526289 Dinh et al. Jun 1996 A
5528409 Cucci et al. Jun 1996 A
5530810 Bowman Jun 1996 A
5533193 Roscoe Jul 1996 A
5533198 Thorson Jul 1996 A
5535326 Baskey et al. Jul 1996 A
5539883 Allon et al. Jul 1996 A
5542055 Amini et al. Jul 1996 A
5546272 Moss et al. Aug 1996 A
5548712 Larson et al. Aug 1996 A
5555510 Verseput et al. Sep 1996 A
5559764 Chen et al. Sep 1996 A
5559958 Farrand et al. Sep 1996 A
5559965 Oztaskin et al. Sep 1996 A
5560022 Dunstan et al. Sep 1996 A
5564024 Pemberton Oct 1996 A
5566299 Billings et al. Oct 1996 A
5566339 Perholtz et al. Oct 1996 A
5568610 Brown Oct 1996 A
5568619 Blackledge et al. Oct 1996 A
5572403 Mills Nov 1996 A
5577205 Hwang et al. Nov 1996 A
5579487 Meyerson et al. Nov 1996 A
5579491 Jeffries et al. Nov 1996 A
5581712 Herrman Dec 1996 A
5581714 Amini et al. Dec 1996 A
5584030 Husak et al. Dec 1996 A
5586250 Carbonneau et al. Dec 1996 A
5588121 Reddin et al. Dec 1996 A
5588144 Inoue et al. Dec 1996 A
5592610 Chittor Jan 1997 A
5592611 Midgely et al. Jan 1997 A
5596711 Burckhartt et al. Jan 1997 A
5598407 Bud et al. Jan 1997 A
5602758 Lincoln et al. Feb 1997 A
5606672 Wade Feb 1997 A
5608876 Cohen et al. Mar 1997 A
5615207 Gephardt et al. Mar 1997 A
5621159 Brown et al. Apr 1997 A
5621892 Cook Apr 1997 A
5622221 Genga, Jr. et al. Apr 1997 A
5623625 Thompson et al. Apr 1997 A
5625238 Ady et al. Apr 1997 A
5627962 Goodrum et al. May 1997 A
5628028 Michelson May 1997 A
5630076 Saulpaugh et al. May 1997 A
5631847 Kikinis May 1997 A
5632021 Jennings et al. May 1997 A
5638289 Yamada et al. Jun 1997 A
5644470 Benedict et al. Jul 1997 A
5644731 Liencres et al. Jul 1997 A
5651006 Fujino et al. Jul 1997 A
5652832 Kane et al. Jul 1997 A
5652839 Giorgio et al. Jul 1997 A
5652892 Ugajin Jul 1997 A
5652908 Douglas et al. Jul 1997 A
5655081 Bonnell et al. Aug 1997 A
5655083 Bagley Aug 1997 A
5655148 Richman et al. Aug 1997 A
5659682 Devarakonda et al. Aug 1997 A
5664118 Nishigaki et al. Sep 1997 A
5664119 Jeffries et al. Sep 1997 A
5666538 DeNicola Sep 1997 A
5668943 Attanasio et al. Sep 1997 A
5668992 Hammer et al. Sep 1997 A
5669009 Buktenica et al. Sep 1997 A
5671371 Kondo et al. Sep 1997 A
5675723 Ekrot et al. Oct 1997 A
5680288 Carey et al. Oct 1997 A
5684671 Hobbs et al. Nov 1997 A
5689637 Johnson et al. Nov 1997 A
5696895 Hemphill et al. Dec 1997 A
5696899 Kalwitz Dec 1997 A
5696949 Young Dec 1997 A
5696970 Sandage et al. Dec 1997 A
5704031 Mikami et al. Dec 1997 A
5708775 Nakamura Jan 1998 A
5708776 Kikinis Jan 1998 A
5712754 Sides et al. Jan 1998 A
5715456 Bennett et al. Feb 1998 A
5717570 Kikinis Feb 1998 A
5721935 DeSchepper et al. Feb 1998 A
5724529 Smith et al. Mar 1998 A
5726506 Wood Mar 1998 A
5727207 Gates et al. Mar 1998 A
5732266 Moore et al. Mar 1998 A
5737708 Grob et al. Apr 1998 A
5740378 Rehl et al. Apr 1998 A
5742514 Bonola Apr 1998 A
5742833 Dea et al. Apr 1998 A
5747889 Raynham et al. May 1998 A
5748426 Bedingfield et al. May 1998 A
5752164 Jones May 1998 A
5754797 Takahashi May 1998 A
5758165 Shuff May 1998 A
5758352 Reynolds et al. May 1998 A
5761033 Wilhelm Jun 1998 A
5761045 Olson et al. Jun 1998 A
5761085 Giorgio Jun 1998 A
5761462 Neal et al. Jun 1998 A
5761707 Aiken et al. Jun 1998 A
5764924 Hong Jun 1998 A
5764968 Ninomiya Jun 1998 A
5765008 Desai et al. Jun 1998 A
5765198 McCrocklin et al. Jun 1998 A
5767844 Stoye Jun 1998 A
5768541 Pan-Ratzlaff Jun 1998 A
5768542 Enstrom et al. Jun 1998 A
5771343 Hafner et al. Jun 1998 A
5774645 Beaujard et al. Jun 1998 A
5774741 Choi Jun 1998 A
5777897 Giorgio Jul 1998 A
5778197 Dunham Jul 1998 A
5781703 Desai et al. Jul 1998 A
5781716 Hemphill et al. Jul 1998 A
5781744 Johnson et al. Jul 1998 A
5781767 Inoue et al. Jul 1998 A
5781798 Beatty et al. Jul 1998 A
5784555 Stone Jul 1998 A
5784576 Guthrie et al. Jul 1998 A
5787019 Knight et al. Jul 1998 A
5787459 Stallmo et al. Jul 1998 A
5787491 Merkin et al. Jul 1998 A
5790775 Marks et al. Aug 1998 A
5790831 Lin et al. Aug 1998 A
5793948 Asahi et al. Aug 1998 A
5793987 Quackenbush et al. Aug 1998 A
5794035 Golub et al. Aug 1998 A
5796185 Takata et al. Aug 1998 A
5796580 Komatsu et al. Aug 1998 A
5796934 Bhanot et al. Aug 1998 A
5797023 Berman et al. Aug 1998 A
5798828 Thomas et al. Aug 1998 A
5799036 Staples Aug 1998 A
5799196 Flannery Aug 1998 A
5801921 Miller Sep 1998 A
5802269 Poisner et al. Sep 1998 A
5802298 Imai et al. Sep 1998 A
5802305 McKaughan et al. Sep 1998 A
5802324 Wunderlich et al. Sep 1998 A
5802393 Begun et al. Sep 1998 A
5802552 Fandrich et al. Sep 1998 A
5802592 Chess et al. Sep 1998 A
5803357 Lakin Sep 1998 A
5805804 Laursen et al. Sep 1998 A
5805834 McKinley et al. Sep 1998 A
5809224 Schultz et al. Sep 1998 A
5809256 Najemy Sep 1998 A
5809287 Stupek, Jr. et al. Sep 1998 A
5809311 Jones Sep 1998 A
5812748 Ohran et al. Sep 1998 A
5812750 Dev et al. Sep 1998 A
5812751 Ekrot et al. Sep 1998 A
5812757 Okamoto et al. Sep 1998 A
5812858 Nookala et al. Sep 1998 A
5815117 Kolanek Sep 1998 A
5815647 Buckland et al. Sep 1998 A
5815652 Ote et al. Sep 1998 A
5821596 Miu et al. Oct 1998 A
5822547 Boesch et al. Oct 1998 A
5835719 Gibson et al. Nov 1998 A
5835738 Blackledge, Jr. et al. Nov 1998 A
5838932 Alzien Nov 1998 A
5841964 Yamaguchi Nov 1998 A
5841991 Russell Nov 1998 A
5845061 Miyamoto et al. Dec 1998 A
5845095 Reed et al. Dec 1998 A
5852720 Gready et al. Dec 1998 A
5852724 Glenn, II et al. Dec 1998 A
5857074 Johnson Jan 1999 A
5857102 McChesney et al. Jan 1999 A
5864653 Tavellaei et al. Jan 1999 A
5864713 Terry Jan 1999 A
5867730 Leyda Feb 1999 A
5875307 Ma et al. Feb 1999 A
5875308 Egan et al. Feb 1999 A
5875310 Buckland et al. Feb 1999 A
5878237 Olarig Mar 1999 A
5878238 Gan et al. Mar 1999 A
5881311 Woods Mar 1999 A
5884027 Garbus et al. Mar 1999 A
5889965 Wallach et al. Mar 1999 A
5892898 Fujii et al. Apr 1999 A
5892928 Wallach et al. Apr 1999 A
5898846 Kelly Apr 1999 A
5898888 Guthrie et al. Apr 1999 A
5905867 Giorgio May 1999 A
5907672 Matze et al. May 1999 A
5909568 Nason Jun 1999 A
5911779 Stallmo et al. Jun 1999 A
5913034 Malcolm Jun 1999 A
5922060 Goodrum Jul 1999 A
5930358 Rao Jul 1999 A
5935262 Barrett et al. Aug 1999 A
5936960 Stewart Aug 1999 A
5938751 Tavallaei et al. Aug 1999 A
5941996 Smith et al. Aug 1999 A
5964855 Bass et al. Oct 1999 A
5983349 Kodama et al. Nov 1999 A
5987554 Liu et al. Nov 1999 A
6009535 Halligan et al. Dec 1999 A
6012130 Beyda et al. Jan 2000 A
6145089 Le et al. Nov 2000 A
6163853 Findlay et al. Dec 2000 A
6170028 Wallach et al. Jan 2001 B1
6173346 Wallach et al. Jan 2001 B1
6179486 Wallach et al. Jan 2001 B1
6192434 Wallach et al. Feb 2001 B1
6219734 Wallach et al. Apr 2001 B1
6247080 Wallach et al. Jun 2001 B1
6247099 Skazinski et al. Jun 2001 B1
6272648 Findlay et al. Aug 2001 B1
6304929 Wallach et al. Oct 2001 B1
Foreign Referenced Citations (5)
Number Date Country
0 866 403 Sep 1998 EP
04 333 118 Nov 1992 JP
05 233 110 Sep 1993 JP
07 093 064 Apr 1995 JP
07 261 874 Oct 1995 JP
Non-Patent Literature Citations (26)
Entry
SPARCstorage Array User's Guide—Sep. 1996, Chapter 8, Getting Started.*
Plasmon M-Series M52, M104, M156, and M258 Optical Disk Library Systems, SCSI Software Interface Specification, Document No. 303432C, Nov. 1996.*
Institute of Information & Computer Sciences, scsi-faq/part1, last modified Mar. 31, 1998.*
Cmasters, Usenet post to microsoft.public.windowsnt.setup, Aug. 1997, “Re: FDISK switches”.
Compaq Computer Corporation, Phenix Technologies, LTD, and Intel Corporation, specification, 55 pages, May 5, 1994, “Plug & Play BIOS Specification”.
Davis, T., Usenet post to alt.msdos.batch, Apr. 1997, “Re: Need help with automating FDISK and FORMat . . . ”.
Davis, T, Usenet post to alt.msdos.programmer, Apr. 1997, “Re: How do I create and FDISK batch file?”.
Gorlick, M., Conf. Proceedings: ACM/ONR Workshop on Parallel and Distributed Debugging, pp. 175-181, 1991, “The Flight Recorder: An Architectural Aid for System Monitoring”.
Hildebrand, N., Usenet post to comp.msdos.programmer, May 1995, “Re: Structure of disk partition into”.
IBM Technical Disclosure Bulletin, 92A+62947, pp. 391-394, Oct. 1992, Method for Card Hot Plug Detection and Control.
Lewis, L., Usenet post to alt.msdos.batch, Apr. 1997, “Re: Need help with automating FDISK and FORMAT”.
Lyons, Computer Reseller News, Issue 721, pp. 61-62, Feb. 3, 1997, “ACC Releases Low-Cost Solution for ISPs”.
M2 Communications, M2 Presswire, 2 pages, Dec. 19, 1996, “Novell IntranetWare Supports Hot Pluggable PCI from NetFRAME”.
NetFrame Systems Incorporated, Doc. No. 78-1000226-01, pp. 1-2, 5-8, 359-404, and 471-512, Apr. 1996, “NetFrame Clustered Multiprocessing Software: NW0496 DC-ROM for Novel® NetWare® 4.1 SMP, 4.1, and 3.12”.
Netframe, http://www.netframe-support.com/technology/datasheets/data.htm, before Mar. 1997, “Netframe ClusterSystem 9008 Data Sheet”.
PCI Hot-Plug Specification, Preliminary Revision for Review Only, Revision 0.9, pp. i-vi, and 1-25, Mar. 5, 1997.
Rigney, PC Magazine, 14(17): 375-379, Oct. 10, 1995, “The One for the Road (Mobile-aware capabilities in Windows 95)”.
SES SCSI-3 Enclosure Services, X3T10/Project 1212-D/Rev 8a, pp. i, iii-x, 1-76, and I-1 (index), Jan. 16, 1997.
Shanley, and Anderson, PCI System Architecture, Third Edition, Chapter 15, pp. 297-302, Copyright 1995, “Intro To Configuration Address Space”.
Shanley, and Anderson, PCI System Architecture, Third Edition, Chapter 16, pp. 303-328, Copyright 1995, “Configuration Transactions”.
Shanley, and Anderson, PCI System Architecture, Third Edition, p. 382, Copyright 1995.
Simos, M., Usenet post to comp.os.msdos.misc, Apr. 1997, “Re: Auto FDISK and FORMAT”.
Sun Microsystems, Part No. 802-6569-11, Release 1.0.1, Nov. 1996, “Remote Systems Diagnostics Installation & User Guide”.
Sun Microsystems Computer Company, Part No. 802-5355-10, Rev. A, May 1996, “Solstice SyMON User's Guid”.
Wood, M. H., Usenet post to comp.os.netware.misc, Aug. 1996, “Re: Workstation duplication method for WIN95”.
ftp.cdrom.com/pub/os2/diskutil/, PHDX software, phdx.zip download, Mar. 1995, “Parallel Hard Disk Xfer”.
Provisional Applications (1)
Number Date Country
60/046327 May 1997 US