1. Field of Invention
The present disclosure relates generally to multi-tasking multi-processor environments, and in particular, to managing recovery and control of a communications link via out-of-band signaling.
2. Description of Background
When operating a communications link in a multi-tasking multi-processor environment, numerous failures can occur and there are a variety of ways in which the communication link can be recovered. For example, in the existing coupling technologies for a multitasking multi-processor environment, each channel is a separate real hardware connection. In these environments, there is hardware support for handling operations on either the channel or the buffer connection level. Operations of this sort include handling timed out or canceled operations. If there is a problem with the entire channel, then the entire link can be taken through recovery with hardware assists.
In the case of the new coupling technology based upon industry standard Infiniband, multiple channels are emulated across a single physical link, and therefore, there is no hardware assist that can be called upon to aid in the recovery of a failed communication link. Further, the data flow design uses pre-defined sequences of resources made available to send and receive messages across any individual virtual connection. If an unexpected message is received, it causes the connection to go into error. This prevents sending a cancel message on the individual buffer connection. Similar problems exist in the handling of channel operations.
An exemplary embodiment includes a computer program product for managing recovery and control of a communication link in a multi-tasking multi-processor environment, the computer program product including a tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method including sending a command, sending an invalidate request to a buffer associated with the command and receiving a response to the invalidate request at least one of prior to the command reaching the recipient and after the command reaching the recipient.
Another exemplary embodiment includes an apparatus for managing recovery and control of a communication link in a multi-tasking multi-processor environment, the apparatus including a processor performing a method including sending a command, sending an invalidate request to a buffer associated with the command, receiving a response to the invalidate request, in response to the invalidate request reaching a recipient prior to the command reaching the recipient, receiving a response to the invalidate request in which a sequence number of the invalidate request mismatched a sequence value saved in a data structure and in response to the invalidate request reaching the recipient after the command reaching the recipient, receiving a response to the invalidate request in which the sequence number of the invalidate request matched the sequence value saved in the data structure.
A further exemplary embodiment includes a method for managing recovery and control of a communication link in a multi-tasking multi-processor environment, the method including sending a command, sending an invalidate request to a buffer associated with the command and receiving a response to the invalidate request at least one of prior to the command reaching the recipient and after the command reaching the recipient.
Other articles of manufacture, apparatuses, and/or methods according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional articles of manufacture, apparatuses, and/or methods be included within this description, be within the scope of the present invention, and be protected by the accompanying claims.
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
The detailed description explains the preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.
In accordance with an aspect of the present invention, the systems, methods and computer program products described herein enable the control and recovery of connections in a multi-tasking multi-processor environment. In exemplary embodiments, the systems, methods and computer program products described herein implement an additional control connection, per channel, across which command and control messages can be sent.
In an exemplary embodiment, a command and control connection can be established over a communications channel. The command and control connection can be implemented to: negotiate the number and sizes of the buffers; manage the establishment of the connections; exchange identification information; handle cancellation of individual operations; assure that the integrity of the channel is preserved; and manage disconnect and recovery of the link.
In an exemplary embodiment, the handling of the command and control connection is generic as compared to the coupling buffer communications connections, since messages can be sent at any time without concern about available receive resources. For example, sufficient resources are in place to handle cancellations for each buffer, heartbeat exchange, and channel recovery operations. In an exemplary embodiment, these resources can be going on in parallel across the same channel connection. Some of the complexity involved in this relates to managing all of these operations in a way that preserves the integrity of all data connections, whether they are on the channel of interest or on other channels running across the same physical link.
In an exemplary embodiment, each of the data connections is managed as a separate Infiniband queue pair. On the recipient side of the primary data connection queue pairs, all required work queue elements (e.g., two) are posted, one for the command and another for the data. The receipt of a third packet can cause the queue pair to go into error, thus preventing further communications across that queue pair. Therefore, the primary originator cannot use this connection to send a cancel message.
In an exemplary embodiment, the auxiliary queue pair can be implemented. A cancel message can be sent across the auxiliary queue pair at any time on behalf of any primary or secondary queue pair, since it maintains a set of receives resources sufficient to handle the maximum number of possible commands. Any request across the auxiliary queue pair can include sufficient information to correlate the request to a specific buffer and operation. The cancel message can specify the particular data queue pair to which the cancel operation is targeted. The cancel message could actually arrive before the recipient received the command and data that is being canceled.
In an exemplary embodiment, in order to isolate the existing multitasking multi-processor environment code from the specifics of this particular coupling technology implementation, the auxiliary queue pair code determines that the work queue elements that are posted on the originator side to receive the response are consumed, which means that the auxiliary queue pair code identifies the current state of processing on the recipient side, and if necessary, executes a portion of the function on behalf of the multitasking multi-processor environment code.
In the case of the buffer invalidate or cancel operation, the systems and methods described herein are implemented under the following conditions: 1) Any buffer for which a cancel operation is begun cannot be released until both the command response and the buffer invalidate response are received and 2) All resources allocated at the transport level must be restored to the expected starting conditions before a new command may start. This condition includes both the send and receive resources for both the originator and recipient sides.
In an exemplary embodiment, the buffer number field (e.g., see
In the case that the buffer invalidate request arrives after the command processing has completed on the recipient side, the auxiliary queue pair support is able to detect this arrival because the sequence number in the invalidate request matches the number stored in the LCB data structure, and the current state indicator indicates that the buffer is ready for new work. As such, the auxiliary queue pair support sends an invalidate buffer response message to the originator without any further processing required, since the command processing has already processed the command and sent the command response.
In the other cases, the sequence number stored in the LCB data structure matches the number in the buffer invalidate request. However, the current state value indicates that the buffer is not ready for new work. In these cases, the auxiliary queue pair support works with the I/O Processor (IOP) to terminate the command processing and ensures that the needs of the originator are satisfied by ensuring that both the response to the command and the response to the invalidate request are sent, and the recipient side receive Work Queue Elements are prepared for receiving the next command.
In multi-tasking multi-processor environments, sending of a command involves sending two messages: a command message and a data message. The recipient side has a matching number of resources. Likewise, for the command response, there must be a matching number of resources between the sending and receiving sides. For response resources, the number of resources depends upon the command, but is known at the time that the command is sent. As such, when the command response is sent either under normal processing or in response to an invalidate request, all of the resources prepared at the time that the command was sent are accounted for in the command response.
This concern is further illustrated in
Further illustration of the invalidate operation is shown in
The invalidate request is by far the most complex of the commands currently managed by the auxiliary queue support. However, other out-of-band messages are frequently exchanged using the out-of-band signaling facility.
Technical effects of exemplary embodiments include the ability to resolve complex issues through the implementation of virtual out of band signaling on a virtual channel, and the components that were used to overcome the complexities associated with the situation.
As described above, embodiments can be embodied in the form of computer-implemented processes and apparatuses for practicing those processes. In exemplary embodiments, the invention is embodied in computer program code executed by one or more network elements. Embodiments include a computer program product 800 as depicted in
While the invention has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include all embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc. do not denote any order or importance, but rather the terms first, second, etc. are used to distinguish one element from another. Furthermore, the use of the terms a, an, etc. do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item.
Priority based on U.S. Provisional Patent Application, Ser. No. 61/031,315, filed Feb. 25, 2008, and entitled “MULTI-TASKING MULTI-PROCESSOR ENVIRONMENTS OVER INFINIBAND” is claimed, the entire contents of which is incorporated by reference in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
3643227 | Smith et al. | Feb 1972 | A |
5170472 | Cwiakala et al. | Dec 1992 | A |
5339413 | Koval et al. | Aug 1994 | A |
5343867 | Shankar | Sep 1994 | A |
5388266 | Frey et al. | Feb 1995 | A |
5524212 | Somani et al. | Jun 1996 | A |
5777987 | Adams et al. | Jul 1998 | A |
6073181 | Holland et al. | Jun 2000 | A |
6289386 | Vangemert | Sep 2001 | B1 |
6483804 | Muller et al. | Nov 2002 | B1 |
6507567 | Willars | Jan 2003 | B1 |
6741552 | McCrosky et al. | May 2004 | B1 |
6862609 | Merkey | Mar 2005 | B2 |
6993032 | Dammann et al. | Jan 2006 | B1 |
7093024 | Craddock et al. | Aug 2006 | B2 |
7200704 | Njoku et al. | Apr 2007 | B2 |
7283473 | Arndt et al. | Oct 2007 | B2 |
7290077 | Gregg et al. | Oct 2007 | B2 |
7366813 | Gregg et al. | Apr 2008 | B2 |
7467402 | Pennington et al. | Dec 2008 | B2 |
7602774 | Sundaresan et al. | Oct 2009 | B1 |
7613183 | Brewer et al. | Nov 2009 | B1 |
20010014954 | Purcell et al. | Aug 2001 | A1 |
20010030943 | Gregg et al. | Oct 2001 | A1 |
20020107903 | Richter et al. | Aug 2002 | A1 |
20020194245 | Simpson et al. | Dec 2002 | A1 |
20030005039 | Craddock et al. | Jan 2003 | A1 |
20030061379 | Craddock et al. | Mar 2003 | A1 |
20040123068 | Hashimoto | Jun 2004 | A1 |
20040202189 | Arndt et al. | Oct 2004 | A1 |
20040221070 | Ortega, III et al. | Nov 2004 | A1 |
20050018669 | Arndt et al. | Jan 2005 | A1 |
20050060445 | Beukema et al. | Mar 2005 | A1 |
20050120237 | Roux et al. | Jun 2005 | A1 |
20050144313 | Arndt et al. | Jun 2005 | A1 |
20060048214 | Pennington et al. | Mar 2006 | A1 |
20060230185 | Errickson et al. | Oct 2006 | A1 |
20060230209 | Gregg et al. | Oct 2006 | A1 |
20060230219 | Njoku et al. | Oct 2006 | A1 |
20070239963 | Yao et al. | Oct 2007 | A1 |
20070245050 | Gregg et al. | Oct 2007 | A1 |
20080028116 | Gregg et al. | Jan 2008 | A1 |
20080109891 | Greenwald et al. | May 2008 | A1 |
20080196041 | Gregg et al. | Aug 2008 | A1 |
20090094603 | Hiltgen et al. | Apr 2009 | A1 |
Number | Date | Country | |
---|---|---|---|
20090216927 A1 | Aug 2009 | US |
Number | Date | Country | |
---|---|---|---|
61031315 | Feb 2008 | US |