Patent Application
20110246677
Embodiments relate to fields of computers systems and, more particular but not exclusively, to methods and systems for handling commands sent to target devices of a computer system. Embodiments also relate to command queuing. Embodiments additional relate to SAS/SATA controllers.
Initiator devices connect computer operating systems or other host systems to network and storage devices. Initiators are the end points that initiate sessions and send commands whereas the targets are the end points that wait for the initiator's commands and provide required input/output (IOs) data transfers. Initiators are coupled to targets via a subsystem or links.
Serial attached SCSI (SAS) and Serial Advanced Technology Attachment (SATA) are some examples of data transfer technologies for moving data to and from target devices including computer storage devices such as hard drives, optical discs, and tape drives. SAS is a serial, point-to-point, enterprise-level device interface that leverages the proven SCSI protocol set. SAS is a convergence of the advantages of SATA II, SCSI, and Fibre Channel, and is the future mainstay of the enterprise and high-end workstation storage markets. SAS offers a higher bandwidth per pin than parallel SCSI and it improves signal and data integrity. The SAS interface uses the proven SCSI command set to ensure reliable data transfers while providing the connectivity and flexibility of point-to-point serial data transfers. The serial transmission of SCSI commands eliminates clock-skew challenges. The SAS interface provides improved performance, simplified cabling, smaller connectors, lower pin count, and lower power requirements when compared to parallel SCSI. SAS controllers leverage a common electrical and physical connection interface that is compatible with Serial ATA technology.
Improved methods and systems are needed for controlling commands to target devices. It is believed that the methods and systems of the illustrative embodiments meet such a need.
The following summary of the invention is provided to facilitate an understanding of some of the technical features related to technique and apparatus for controlling commands for a plurality of target devices and is not intended to be a full description. A full appreciation of the various aspects of the invention can be gained by taking the entire specification, claims, drawings, and abstract as a whole.
The aforementioned aspects of the invention and other objectives and advantages can now be achieved as described herein.
According to one aspect, a method is provided for controlling commands for a plurality of target devices. The method can comprise: setting in circuitry of a hardware controller allowed to queue depths of each one of a plurality of target devices supported by the controller; monitoring status of each one of the plurality of target devices using circuitry of the hardware controller; and, for each one of the plurality of target devices, using circuitry of the hardware controller to control queuing of the commands according to the queue depth setting for the target device.
According to another aspect, a hardware controller is provided for controlling commands for a plurality of target devices. The hardware controller can have queue depth settable circuitry, target monitoring circuitry, and queue management circuitry. The queue depth settable circuitry can be adapted to enable allowed queue depths of each one of a plurality of target devices supported by the controller to be set in the circuitry. The status monitoring circuitry can be adapted to monitor status of each one of the plurality of target devices. The queue management circuitry can be adapted to control queuing of the commands for each one of the plurality of target devices according to the queue depth setting for the target device.
According to yet another aspect, a system is provided for controlling commands for a plurality of target devices. The system can have an initiator and the aforementioned hardware controller is operably coupled to the initiator.
The accompanying figures, in which like reference numerals refer to identical or functionally-similar elements throughout the separate views and which are incorporated in and form a part of the specification, further illustrate the present invention and, together with the detailed description of the invention, serve to explain the principles of the present invention.
The particular values and configurations discussed in these non-limiting examples can be varied and are cited merely to illustrate at least one embodiment of the present invention and are not intended to limit the scope of the invention.
SAS/SATA controllers and other initiator controllers process commands sent by a host operating system which typically is allowed to send many more commands to the controller then the controller can send to a target device. This requires the controller to keep track of the number of commands outstanding to each device (numbers of commands the controller has active to the target device) and if the host sends more commands than allowed by the device or more commands that may hinder overall performance (queue full status), the controller should queue the commands based on a queue depth per target devices that is managed by the controller.
Heretheto now, this type of queue depth management is typically performed by firmware based algorithms running on embedded CPU/s within the controller. In order to increase performance (reduce the firmware command processing time), the controller must support faster and/or multiple CPU's to run the firmware algorithms. This typically increases one or all (cost, power, size, complexity, time to market). Some solutions (controllers) require that the operating system manage the queues so the controller does not have to. These solutions typically require more complex drivers that have to execute more code per command than solution that have the controller do it.
Technical features described in this application can be used to construct various systems and methods for controlling commands for a plurality of target devices. The approach implements functionality in a hardware controller to handle command queuing concurrently on a per “device handle” basis. The hardware controller handles command queue depth control to reduce target queue full exceptions status and to limit command queue depths for devices with limited queue depths such as, for example, SATA NCQ 32, that does not require any firmware processing. This approach can also allow for firmware to determine when to start and stop queuing commands during expectation cases. Also, the approach can enable queued commands to be automatically started after the completion of previous commands and can allow for firmware to start queued commands. A single physical target can have multiple associated device handles where each device handle has separate command queues, queue depths, and initiator address. The hardware can keep state information on a per device basis and adjusts it per command during the start and completion processing.
An example of a computer system architecture including a command queuing hardware controller, according to one embodiment, is illustrated in
The initiator device 102 may be, for example, a serial attached SCSI (SAS) initiator, Serial Advanced Technology Attachment (SATA) imitator, or Serial Tunneling Protocol (STP) initiator. The sub system 104 can comprise direct connections (physical links), expanders, or a combination thereof to link the initiator device 102 to the plurality of target devices 105. The subsystem configuration can vary depending on the number and type of target devices. For example, a SAS initiator may be coupled to a plurality of target devises using one or more SAS expanders. The target devices 105 may be, but not limited to, hard disk drives, optical drives, tape drives, and other types of storage devices.
The command queuing hardware controller 103 may be physically included in the initiator device 102 itself or external to the initiator device 102. The controller 103 can be an ASIC or other hardware circuitry. Referring additionally to
Hardware controller 103 includes queue management logic circuitry 204, configurable queue depths 203, and target monitoring circuitry 205. For each one of the plurality of target devices 105, target monitoring logic circuitry 205 is configured to monitor the status of the target device 105 to keep track of the number of commands outstanding to the target device. Target monitoring circuitry 205 can be configured to monitor state information on a per device basis and per command. Also, for each one of the plurality of target devices 105, queue management logic circuitry 204 controls the number of commands outstanding to the target device by queuing the commands according to an allowed queue depth 203 assigned to the target device. The queue depth is the number of commands a device can hold in its command queue. When the queue is full, the target device will refuse to accept any additional commands. The device will continue to refuse new commands until at least one command has been completed, freeing up space in the queue. The allowed queue depth 203 for each particular device resides in hardware circuitry 202 of the controller as settable flag bits. The queue depth flag bits can be settable by firmware 201. The allowed queue depth value can be settable anywhere between a minimum or maximum value supported by the particular target device.
The queue management logic circuitry 204 can be further configured to automatically start queued commands after completion of previous commands. The firmware 201 can be further configured to determine when to start and stop queuing commands during exception cases and can be configured to start queued commands also.
In one embodiment, the hardware circuitry 202 in controller 103 is adapted to enable the allowed queue depths 203 to be adjustable by the firmware 201. For example, for each target device 105, the allowed queue depth 203 can be dynamically increased or decreased by the firmware 201 adjusting the allowed queue depth bit flags based on status reported back to the initiator by the target. The allowed queue depth 203 can be adjusted per command during the start and completion processing. This improves the per command processing performance of the initiator device.
Method of
As will be explained in more detail below, in one embodiment, a state machine of the hardware controller can be additionally configured to control pending for task management or other exception cases.
Reference will now be made to
The SAS Controller “Fast path Engine” 600 comprises two major hardware blocks, the requestor 605 and completer 604. These hardware blocks work concurrently on processing IO requests from the operating system and forwarding those requests to a SAS initiator protocol engine 611. The completer 604 processes completed successful commands from the SAS protocol engine 611 and completes those commands back to the operating system by way of the messaging unit hardware 602 (PCIe register set interface).
Requestor 605 and completer 604 manage queuing of new command requests from the operating system on a per target device basis. Queuing is controlled by Qdepth programmed within a SAS target device context array 609 (FPE device structure) by the controller firmware. The FPE device structure stores context and control information about the specific target end device. This IO count can be adjusted (Read/Write) at anytime by the controller firmware (FW) through atomic hardware register access located in a FPE register control block 608.
The Fastpath Engine 600 manages the queue depths of each target Device 610, managing a separate linked list of IO requests (commands) on a per target device basis. Target Devices can be, for example, SAS Expanders, SAS end devices, and/or SATA target devices. (See T10 SAS specification).
FPE handles pending IOs for QFull and firmware controlled exception cases. The FPE includes a state machine including command structure. In the specific embodiment of
Each SCSI, SATA device, or other target device (per DevHandle) is classified as fast path capable. Each IO request is classified as fast path capable. A DeviceHandle is an index into the target device context array 609. The firmware determines which devices and/or DevHandles are fast path enabled; this may be determined by the device type when the firmware creates the DevHandle at device discovery time. The host driver determines which IO is Fastpath by selecting a SCSI command request message descriptor type.
The OS driver can choose to send an IO as fast path or normal by building different request descriptors. Note that not all SCSI command request messages will be Fastpath capable, for instance, SATA non R/W commands. Note also that not all device types are Fastpath capable such as IR volumes and some SATA, SES, and ATAPI devices.
FPE requestor 605 processes (is the consumer of) the FPE Request FIFO (process 703). To this end, requestor 605 validates the request is correct and the device is enabled. Requestor 605 can validate the request by validating the SCSI command request message parameter.
The FPE requestor 605 checks per device firmware settings that control operations of the device. The FPE requestor 605 does this by checking FPEDevice FWFlags of the device structure within target device context array 609. The requestor 605 stores (manages) context information about the requests in the FPE Device context data structure. This context information comprises Active loCount, Pending IO Linked list, and HW State Flags. The requestor 605 passes control of IO structure to SAS initiator protocol engine 611 if all tests pass and IO is OK to send to target device 610.
The SAS protocol engine 611 and target device 610 complete IO (process 704). The SAS protocol engine 611 posts successfully completed IO to the FPE Completer completion FIFO 603 (process 705).
The FPE completer 604 processes the completed IO (process 706). To this end, the FPE completer 604 updates the active IOCount and context information of the FPEDevice context structure and the IO context structure. The FPE completer 604 collects information and builds the reply structure and posts this to the Messaging unit hardware to reply back to the operating system via the MU Reply FIFOs.
A command queuing method according, to an embodiment, will now be described in detail with, reference to the system of
Host message unit requests (includes set of virtual function requests queue) is provided (421). A start path and completion path operate concurrently. The start path is initiated by the firmware posting to request queue to start IO (401). The completion path feeds a completion state in response to completion of an IO (402). A list of pending IOs waiting to start is linked to the completion path. When the FPE hardware completes processing of an IO (command) on the completion path, the FPE checks the list to see if any commands are currently pending and starts those next before any new ones are received from the host.
A determination is made as to whether fast path through the FPE hardware is enabled for the target device (403). The Fast Path (FP)_Enable hardware flag is used to indicate if fast path is enabled for the target device associated with the IO. The FP_Enable flag is set in response to the firmware determining if the target device can be processed by the hardware fast path. If fast path is not enabled for the associated target device, the IO fails to start (exception) and the queue feeds to the firmware (404). If the fast path is enabled, a determination is made as to whether an IO request was received from the host operating via the messaging unit (405). If so, a security/access control check is undertaken to determine if the request came from a host queue that the target device is allowed to receive commands from (406) (hardware supports sr-iov, multiple request host queues from different PCIe virtual functions). If not, IO fails to start and queue feeds to firmware (404).
If a request was not received from the host operating via the messaging unit (405), the firmware sets an auto pending flag in the state machine for a given IO (FW start IO, host cannot set this bit) when it wants the hardware to start pending this IO and all following IOs until the firmware clears the bit. If auto pending is not started or is stopped (407) or the security/access control check determines a request came from a host queue that the target device is allowed to receive commands from (406), the firmware also has the option to force pending (set force pending hardware flag) when the firmware requires the hardware to force start pending IO for a given device (408). A common case for SATA device is when a non-NCQ command comes in. All current NCQ commands for the device must be completed before the non-NCQ command is started. While waiting for the NCQ command to complete, all new requests must be pended. The firmware will also set this when it is dealing with SCSI Task Management cases (aborting commands or resetting the target).
If it is determined that pending mode for a given target is set (407), a hardware flag is set when the first IO is pended and cleared when the last IO on the pend list gets started (409). The flag indicates that a new IO needs to be pended at the end of the list. A synchronous event back to the firmware informs the firmware that IOs are now being pended (410). The auto pending mode is set (411) in response. Auto pending is also set if force pending mode is not set (408).
The firmware clears both the pending bits (Auto pending and/or force pending) for non-native command queue (NCQ) command and Task management cases. If the auto pending mode (411) is not set, the process makes a determination as to whether the IO can start based on queue depth (412). If so, instead of having the hardware manage the pending IO, it can be sent to the firmware to have the firmware manage the IO (413). If the IO is either sent to firmware to manage IO (414), force pending mode is set (408), or the auto pending mode (411) is set, a determination is made as to whether the IO is already on the pending list (414). If the IO is already on the pending list, the pending IO cannot start so the IO remains at the head of the pending list (415). If the IO is not already on the pending list (414), a pending process is implemented in which the IO is pended (417). A hardware bit is set to indicate that the IO is being pended (418). The IO is fed to the FPE engine pend queue (419).
If IO is not sent to firmware to have firmware manage the IO (413), the IO fails to start (exception) (404). If the IO cannot be started based on qdepth (412), after starting one IO, the next pend IO on this list is started (416). This keeps the state machine in this “start pending” state until it either starts all pending IOs or reaches qdepth limit. If the state machine can start all pending IOs (416), the process turns to start path (401). Otherwise, process is done.
In yet another embodiment, controlling of the queue depths for respective target devices according to the aforementioned methods and systems of the embodiments can be utilized to provide different priority and quality of service for different IOs flows to target devices. Reference will now be made to
If the queue depth setting 203 for the first target device A is higher than the second target device B, the hardware controller 103 will deliver more concurrent commands to first target device A than to the second target device B. Similarly, if the queue depth setting 203 for the first target device A is lower than the second target device B, the hardware controller 103 will deliver more concurrent commands to the second target device B than to the first target device A. Thus, the hardware controller delivers more concurrent commands and therefore priority to whichever one of the first and second target devices has the higher queue depth setting (506). This means that priority and quality of service for IOs to different target devices can be controlled by adjusting the allowed queue depths settings 203 for the different target devices (within the maximum and minimum values supported by the target devices). Controlling the queue depths of the plurality of targets to make one or more particular target devices have more concurrent IOs then the others results in the particular target device(s) having slightly higher priority than the others and the controller delivering more bandwidth to the operating system for that particular target or set of targets. The method of
The system and methods of the illustrative embodiments handle multiple devices concurrently and track the number of outstanding commands to each device and the queue depth control of each device independently. This can eliminate firmware processing of commands by the controller when the command queue depth handling is required based on per device queue depths which reduces the overall per command processing time by the controller. The systems and methods of the embodiments greatly improve the overall per IO performance of the controller without having to increase the CPU clock frequency or to add multiple CPU cores. This solution reduces command processing overhead compared to a solution that requires a driver to handle the command queuing. Eliminating CPU assistance from the normal IO path can reduce CPU performance requirements, reduce, and align arbitrated memory accesses to improve performance of memory subsystems, and possibly allow use of smaller and more energy efficient designs.
The embodiments and examples set forth herein are presented to best explain the present invention and its practical application and to thereby enable those skilled in the art to make and utilize the invention. Those skilled in the art, however, will recognize that the foregoing description and examples have been presented for the purpose of illustration and example only.
Other variations and modifications of the present invention will be apparent to those of skill in the art, and it is the intent of the appended claims that such variations and modifications be covered.
The description as set forth is not intended to be exhaustive or to limit the scope of the invention. Many modifications and variations are possible in light of the above teaching without departing from the scope of the following claims. It is contemplated that the use of the present invention can involve components having different characteristics.
