Patent Application
20120026863
Performance improvements in computing and storage, along with a drive to exploit these improvements in highly challenging applications, have increased the demand for extremely fast data links, for example in areas of high-speed and data-intensive networking. One example of a highly challenging application is data replication in information storage and retrieval. For systems that are expected to operate continuously, a duplicate and fully operational backup capability is implemented in the event a primary system fails.
Data replication, which may also be called mirroring, is a process that generates a mirrored copy of information on two or more storage devices, a primary volume and one or more secondary volumes. The goal is availability of important information on duplicate storage devices at all times. The duplicates may reside on the same or different devices or systems. Similarly, the duplicates may reside on local or remote devices or systems. The obvious advantage of remote replication is avoidance of destruction of both the primary and secondary copies in the event of a disaster occurring in one location.
Corporations, institutions, and agencies sharing common databases and storage systems often include enterprise units that are widely dispersed geographically and therefore may use data replication over very large distances. Distance between storage sites increases communication latency, and reduces speed and reliability, although the demand for fast communication remains.
In response to the demand for fast data links, various network interconnect standards have been developed to enable faster communication between computers and input/output devices. One example of an interconnect standard is a Fibre Channel (FC) standard and associated variants, which are defined in an effort to facilitate data communication, including network and channel communication, between and among multiple processors and peripheral devices. The Fibre Channel standard enables transfers of large information amounts at very high rates of two or more gigabits (Gb) per second.
Remote replication links in storage systems tend to be exclusively standard links with a specified standard throughput, for example 1-2 Gb for the Fibre Channel standard. Replication links may be implemented on other standards, such as Enterprise Systems Connection (ESCON), Small Computer Systems Interface (SCSI), and others.
A communication method comprises communicating information on a standardized link compliant with a published standard and communicating information on a private link capable of operation at a throughput higher than maximum throughput of the published standard. The method further comprises monitoring private link performance and increasing information throughput on the private link above the maximum standard throughput by an amount determined by the monitored private link performance.
Embodiments of the invention relating to both structure and method of operation, may best be understood by referring to the following description and accompanying drawings:
For a network or interconnect system or for a geographically dispersed storage or database system that implements industry standard components, devices, and other infrastructure elements, communication rates are limited to the throughput specified by the standard. In various applications, configurations, and conditions, the communication links connecting particular networked devices may have a length much less than the maximum length defined by the standard. Similarly, the communication links may have a signal quality much higher than the minimum specified for the standard.
An interconnect structure or network may include one or more nonstandard links and exploit the nonstandard character of the link by disregarding constraints to operate according to industry standard throughput rates, although the communication may follow protocols similar or identical to industry standard protocols. By disregarding the data rate constraints, an illustrative system can substantially increase throughput. In various embodiments, the nonstandard links may be private links, dark fiber links, dark fiber point-to-point links, or other links.
Standards are set according to attainable technological capabilities in typical operating conditions and configurations. Standards for communication links are accordingly set to handle communications between sites separated by hundreds of miles with limited signal quality. The disclosed structures and communication techniques acknowledge that many, in fact most, communication links are substantially shorter and have substantially higher signal quality than the “least common denominator” to which the standards are set. The disclosed structures and communications techniques exploit the expected more favorable communication conditions, when appropriate, to enhance throughput. Accordingly, the disclosed structures and communication techniques generally produce a significant throughput improvement. In many embodiments, the disclosed structures and communication techniques include support for the standard techniques so that throughput in the system disclosed herein is at least as good as the standard implementation.
Referring to
A private link 104 is defined herein as a communication link that is exclusive of industry standard links that operate at an industry standard throughput rate. A common example of an industry standard communication link is the Fibre Channel standard. In some embodiments, the private link 104 may be further defined as a point-to-point communication link between network devices, such as storage arrays 106. In various embodiments, a private link 104 may be further defined as a communication link that does not have one terminus directly attached to a public, shared network which supports a standard managed network. Although a signal may transit a private link and ultimately reach a standard network link, the private link retains character as a private link. In some embodiments, private links 104 may be constructed with dedicated leased lines, dial-up lines, satellite links, microwave links, or the like. The private links 104 may connect local sites or remote sites. Private links 104 are typically considered private on the basis that no other traffic than traffic of the entity leasing or controlling the private links communicates on the links.
Private networks have traditionally been built with dedicated leased lines, dial-up lines, or other links such as satellite or microwave. Links are established among remote sites. The links are “private” because no other traffic except the traffic of the company leasing the links crosses the links
In a typical embodiment, the communication controller 102 is adapted to reside within storage arrays and communicate information on a point-to-point remote replication link 108 interconnecting one or more storage arrays 106. In some embodiments, the communication controller 102 manages information communication information on a private dark fiber point-to-point link connecting remotely replicated storage arrays 106. In some embodiments, a point-to-point link may use a specially-modified Dense Wavelength Division Multiplexing (DWDM) extender which has a capability to adjust to varying bit times.
Dark fiber is defined as an optical fiber infrastructure, including cables and repeaters that are currently installed and existing, but is not currently used in a public network. A sometimes substantial amount of dark fiber-optic cable may exist because the cost of installing cable in excess of expected short-term capacity is small in comparison to installing more cable at a later time. The dark fiber may be leased for usage by individuals or small operations and thus may be neither controlled nor connected to a public utility. Accordingly, the small operation or even an individual may supply components sufficient to make the dark fiber operational. The dark fiber may further be used to interconnect relatively local components of a storage system.
In various embodiments, the storage apparatus 100 is an adapter, device, component, or the like that facilitates and enhances communication. Common examples of the storage apparatus 100 are Network Interface Cards (NICs), Transmission Control Protocol/Internet Protocol (TCP-IP) NICs, Host Bus Adapters (HBAs), Storage Area Network (SAN) HBAs, network management appliances, intelligent switches, or any similar functional element. The storage apparatus 100 is commonly used to offload input/output operations from a computing element such as a host computer so that data transfer rates increase. Although TCP-IP Network Interface Cards (NICs) and SAN Host Bus Adapters (HBAs) vary greatly in the amount of attainable host offloading, a generally common characteristic of standard input/output interfaces, regardless of communication protocol, is that a maximum theoretical distance, and corresponding minimum per-bit clock rate, is de-rated by as much as 20-100% to account for wide variability in link type, link quality, fiber bend radius, cross-talk, and poor mechanical connections.
The effect of the lowering the rated optical transmission capability due to such deterioration or insufficiency is that the published interface standard is highly compromised to account for compatibility under worst case or least common denominator conditions. For example, the interface standard of 2 gigabits (Gb) per second for Fibre Channel may, under best-case conditions, easily enable a throughput of two to ten times the bit rate of the published specification.
Under industry standard constraints, a private link, without intermediate commercial repeaters or switches, is commonly used as the weakest link in what may be a multiple million dollar enterprise data replication scheme. The cost of leased dark fibers over time may overwhelm the original hardware cost. Accordingly, the illustrative apparatus 100 and associated operating method may enable a tremendous competitive advantage, possibly a cost advantage in the range of millions of dollars, over several years.
In some embodiments, the communication controller 102 may be adapted to communicate information on the private link 104 using a communication technique selected from among multiple communication techniques that are implemented and operable on the private link 104. In a typical embodiment, the communication controller 102 supports communication techniques operable on the private link 104 including an industry standard communications technique and an increased throughput communications technique.
In a particular embodiment, the apparatus 100 may further comprise a bit error rate analyzer 110 coupled to the communication controller 102. The communication controller 102 can selectively communicate information on the private link 104 using a communication technique that increases throughput rate while balancing bit rate and bit error rate.
In some embodiments, the communication controller 102 is adapted to communicate information on the private link 104 using a digital pulse width modulation communication technique that encodes information as an increment in pulse width over a nominal pulse width.
Other embodiments may combine multiple techniques, for example using both the bit rate analyzer 110 and digital pulse width modulation. Accordingly, throughput is increased by selectively communicating information on the private link 104 by balancing bit rate and bit error rate, and throughput is further enhanced by the digital pulse width modulation technique.
The particular communication technique that executes at a particular time can be determined using various techniques. In some conditions, implementations, and/or embodiments, the apparatus 100 includes a link analyzer 112 that is connected to the private link 104 and operates in conjunction with the communication controller 102. The link analyzer 112 determines a condition of communications on the private link 104. The communication controller 102 selects from among the multiple communication techniques based on the communications condition.
In some conditions, implementations, and/or embodiments, the apparatus 100 further comprises a graphical user interface 114 that interacts with the communication controller 102 and is operated by a user to select from among the multiple communication techniques.
Referring to
In various embodiments and/or in particular conditions, the selection may be made either under external control or automatically. For example in some embodiments, a graphical user interface (GUI) may be implemented to enable selection of an operational communication technique from among the techniques of the multiple implemented tiers. In some embodiments, the communication technique may be automatically determined according to results of information sensed as the link communicates information. Some embodiments may include both the GUI support and sensors that enable automatic communication technique selection.
In a particular embodiment, a Tier-0 technique may enable usage of the standard interface, for example a Fibre Channel interface, without alteration. The communication controller 202, based on selection of a communication technique, activates the communication technique selected from among the industry standard communications technique and one of the increased-throughput communication techniques.
In an example configuration, the Tier-0 technique may activate an industry-standard host bus adapter (HBA) and possibly industry-standard switches and/or repeaters without modification so that the bit rate native to the industry-standard components is implemented without rate modification. The apparatus 200 includes a graphical user interface (GUI) 206 and a throughput control logic 208, for example residing on a throughput control-enabled host bus adapter 200. During high-throughput operation with non-standard usage of the protocol's bit rate active, for example with either Tier-1, Tier-2, or Tier-3 operating, a user may revert back to Tier-0 operation, thereby changing communication bit rate to the standard rate.
An embodiment may implement Tier-1 as a logic that enables a decrease in 1/0 signal bit time by increasing the bit clock rate, for example by an increase of one to ten times the standard bit clock rate, for links which are shorter and/or having higher signal quality than a worst case condition.
Some implementations may configure Tier-2 to include a suitable type of signal data compaction. A typical throughput increase resulting from Tier-2 operation may be in a range of four, eight, sixteen or more times normal throughput in an illustrative example whereby the communication controller 202 uses digital pulse width modulation to encode information as an increment in pulse width over a nominal pulse width. Digital pulse width modulation can be implemented in a manner that stretches the bit time and interprets the increase in width as information content of four bits (a nibble), a byte (eight bits), a word (sixteen bits), or more.
In a general digital pulse width modulation application, a pulse with a defined nominal pulse width is extended an interval based on a digital value to be communicated. The number of secondary clock cycles that the extended pulse width exceeds the nominal pulse width is counted. The count corresponds to the digital value. Various embodiments use a counter to determine the extent that the modulated pulse width exceeds the nominal pulse width.
Pulse width encodes the digital value communicated and counters are used to extract the digital data. The digital pulse width modulator does not use a carrier but instead uses synchronized transmitter and receiver clocks running at substantially the same frequency so that the amount of time the modulated pulse is extended is measurable. The digital pulse width modulator performs data compaction by representing either a greater number of bits in the same time frame or the same number of bits in a smaller reference frame.
The illustrative storage apparatus 200 is depicted in the form of a storage controller or possibly a host bus adapter, The apparatus may include a plurality of embedded processors 210 generally are high-performance processors that are capable of transferring information at a high rate to support multiple storage devices in a scaleable storage array controller. A memory controller 212 may be connected to the embedded processors 210 and operates as a hub device to transfer data point-to-point or, in some embodiments on a network fabric, among the multiple storage levels. The illustrative memory controller 212 has multiple channels for communicating with a cache memory 222 to ensure sufficient bandwidth for data caching and program execution. The memory controller 212 has sufficient performance to manage the multiple I/O channels 218.
An Ethernet interface 216 communicates with the memory controller 212 via the communication controller 202 which may be an input/output (I/O) controller hub that includes an integrated Fast Ethernet Media Access Controller (MAC) to form a local area network (LAN) management interface port. The I/O controller hub 214 includes typical peripheral interfaces including Universal Serial Bus (USB), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE), General Purpose Input/Output (GPIO), System Management Bus (SMBus), and the like.
The link 204 may connect to a switch fabric interconnect device 220 and a controller such as a Gigabit Ethernet or Fibre Channel controller based upon the type of network fabric, iSCSI or Fibre Channel.
Referring to
Referring to
Digital data is communicated by modulating the pulse train waveform 312 to produce a pulse width modulated waveform 316. The modulation effectively shifts the falling edge of the modulated pulse.
In one embodiment, the nominal pulse width is only increased, not decreased, to encode the digital information. The pulse width is increased only in discrete amounts such that the pulse falling edge may occur at n possible locations including the position of the falling edge of the nominal pulse. In one embodiment, the maximum modulated pulse width is twice the nominal pulse width.
The pulse rising edge indicates transmission of a new digital data value. The pulse falling edge indicates the end of transmission of the digital data value and is determinative of the value. The data value may be recovered by determining the amount the modulated pulse width exceeds the nominal pulse width. In one embodiment, a demodulation counter determines the data value. The counter effectively functions as a digital integrator but is unaffected by amplitude, phase, or frequency of the modulated signal, other than triggering thresholds that start and stop the counter. The digital value is indicated only by the change in nominal width, rather than the total pulse width. Therefore the counter is to account for the nominal pulse width. One embodiment accounts for the nominal pulse width by integrating or counting throughout the entire pulse width and subtracting the nominal pulse width count. Another embodiment resets the counter or configures the counter to rollover at the count associated with the nominal pulse width count.
A Tier-3 implementation may combine Tier-1 and Tier-2 concurrently to attain a shared effect in the illustrative example of four to eighty times the standard throughput. Tier-3 is typically effective for links that are much shorter and/or having much higher signal quality than the limits imposed by the standard. Typically, conditions enabling Tier-3 operation are attainable for links that are confined to a single electronics system rack or to a single data center. The Graphical User Interface (GUI) 206 may be used to activate Tier-3 operation based, for example, on knowledge of link length and conditions. The GUI 206 may also access and display information from an attached bit rate tester or other link analyzer to enable the user to determine whether Tier-3 operation is appropriate.
In some embodiments, Tier-3 may include dynamically-selected compaction whereby the increase in throughput rate of Tier-2 data compaction is selected according to line quality, as determined by the bit-error-rate measured in Tier-1 operation.
Referring to
The communication controller 402 may increase the bit rate, for example in a range from one to ten times the standard rate, for a communication link that is somewhat shorter and higher quality than the worst case link under the industry standard. An example of a higher quality communication link may be a combined optical/copper cable with a relatively small number of mechanical connectors or an optical cable with small radius bends, or a copper cable.
In an implementation example, a graphical user interface (GUI) 408 on a host computer may be operational, following connection of host bus adapters and communication links in operational positions, in a link throughput test mode. The GUI 408 may include a slider bar which enables optimization of either bit-rate or bit-error-rate in two different operating modes.
In a first mode, a user can select a GUI setting to enable a free-floating bit-error-rate and a fixed bit-rate. The user sets the slider bar to an arbitrary bit-rate setting within an allowable throughput enhancement range. A typical enhancement range may be one to ten times the standard throughput, although any suitable enhancement may be implemented. The bit-error-rate analyzer 406 continuously runs and reports a real-time bit-error-rate. For example, a sample test pattern may be transmitted back and forth across the link or links in a bi-directional manner while the bit-error-rate is calculated in real-time. The slider bar may be left at a maximum bit rate at which an acceptable bit-rate-error can be empirically attained. For example, one bit error in 1010 bits may be defined as a low standard. One bit error in 1015 bits may be specified as an average standard. One bit error in 1017 bits may be assigned to a high standard. The time to send such a large number of test bits over a link is too lengthy to be feasible. A more realistic and typical empirical approach is to continually reduce the bit time until a first error occurs, then reduce the bit-rate by a de-rating factor of 10-30%.
In a second mode, the user may select a GUI setting to enable a free-floating bit-rate and a fixed bit-error-rate. Based on the specified maximum bit-error-rate, the real-time bit-error-rate analyzer 406 detects the error-rate and the bit-rate is automatically and continuously adjusted, incremented to decremented, to hold the bit-error-rate at the specified maximum bit-error-rate. The user may select the maximum bit-error-rate from a menu of typical values in a range from ideal to poor.
In some implementations or embodiments, the fixed bit-error-rate may be dynamically determined, for example in response to demand, so that higher error rates may be tolerated if the demand is uniquely met by other aspects of the system. For example, a system may have additional error detection capability to reduce or minimize the number of undetected errors. Internet Small Computer Systems Interface (iSCSI) has a suitable error detection capability. Demand may be determined by sensing of various conditions such as throughput, queuing, and the like.
During typical operation, once a user makes selections via the GUI 408, and selects possible de-rating factors such as 10%, 20%, or the like, the throughput control maintains a substantially steady-state with no change to initialized parameters unless a physical change affects the link, such as a degraded or broken physical connection, a cable bend radius is altered, or the like. In the event the standard protocol, for example Fibre Channel protocol, encounters error and re-tries, the throughput enhancement controller-enabled devices, for example host bus adapter, may be enabled to make automatic adjustments until the re-tries no longer occur.
A typical apparatus implementation may include the bit-error-rate analyzer 406 and other link analysis capabilities, for example by generating continuous data or control characters on a transmit path and expect continuous data or control characters on the receive path. Monitoring and analysis of the received data enables link fault detection and generation of signals indicating reliability of the link. The apparatus 400 may implement test pattern generation and checking for link testing and system testing. The bit-error-rate analyzer 406 typically performs monitoring and generates an error rate indication to enable constant line quality monitoring.
In a typical embodiment, the bit-error-rate analyzer 406 is transparent to the link and generates bit error rate patterns using American National Standards Institute (ANSI) standards. The patterns are framed using suitable framing protocols so that the link is tested with actual data traffic at actual operating speeds of the data link. Bit errors are counted and used to supply a statistical measurement of data link quality.
In a particular implementation, a pair of Fibre Channel interface cards may be equipped with throughput enhancement devices and used to facilitate communications within a single electronics rack, for example with an approximately one meter cable, to operate at a data rate of four to eighty times the standard 1-2 Gb throughput rate. A similar connection within a single data center, for example with an approximately ten meter cable, may operate at a data rate of four to ten times throughput improvement. Even a link with a multiple-kilometer length dark fiber, for example a premium optical fiber in a private trench, may realize a two times or more improvement in industry-standard throughput.
Throughput of long distance replication links is generally the single most limiting factor—the weakest link—in a multiple million dollar data replication arrangement. Accordingly, implementation of the illustrative throughput enhancement may dramatically improve the total number of links implemented in a system, the initial copy times for data volumes, and the maximum traffic per link. A reduction in the total number of implemented links may save substantial expense which, over time, may eclipse the cost of the storage arrays in the replication system.
For short-distance links, threshold enhancement-equipped Fiber Channel interface cards may be used as an alternative to infini-band and other high bandwidth link types.
While the present disclosure describes various embodiments, these embodiments are to be understood as illustrative and do not limit the claim scope. Many variations, modifications, additions and improvements of the described embodiments are possible. For example, those having ordinary skill in the art will readily implement the steps necessary to provide the structures and methods disclosed herein, and will understand that the process parameters, materials, and dimensions are given by way of example only. The parameters, materials, and dimensions can be varied to achieve the desired structure as well as modifications, which are within the scope of the claims. Variations and modifications of the embodiments disclosed herein may also be made while remaining within the scope of the following claims. For example, the disclosed apparatus and technique can be used in any storage and communication configuration with any appropriate number of storage arrays or elements. The various communication controller, analyzer, and bit test elements may be implemented in any suitable component or device, for example host computers, host bus adapters, storage controllers, disk controllers, management appliances, and the like. The links may be in any configuration and have lengths of any suitable distance. Although, the illustrative system discloses magnetic disk storage elements, any appropriate type of storage technology may be implemented. Control elements may be implemented as software or firmware on general purpose computer systems, workstations, servers, and the like, but may be otherwise implemented on special-purpose devices and embedded systems.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11027170 | Dec 2004 | US |
| Child | 13246504 | US |
