Patent Grant
8196139
This invention relates, in general, to input/output (I/O) processing, and, in particular, to facilitating interpretation of I/O operation requests of pageable guests.
Efficient input/output (I/O) processing is an ever-increasing challenge, and strides are continuously being made towards improving efficiency. As one example, in order to achieve considerably higher bandwidths than previously allowed, an enhanced input/output architecture, referred to as the queued direct input/output (QDIO) architecture, has been designed by International Business Machines Corporation. This architecture is exploited in various systems and products, including the zSeries systems (e.g., z900) and the z/VM operating system products offered by International Business Machines Corporation. These systems and products have been enhanced to allow multiple operating system images, operating under host (e.g., z/VM) control, to perform queued direct I/O operations using TCP/IP communications adapters and Fibre Channel Protocol (FCP) adapters.
Currently, use of the QDIO architecture by guest operating systems executing in pageable mode virtual machines requires extensive overhead by the host. For instance, the host needs to create and maintain shadow QDIO data queues, which mirror the QDIO actions taken against the guest operating system created QDIO queues used to access the network or I/O devices attached to the QDIO adapters. This QDIO shadowing/mirroring requirement exists due to the intrinsic manner by which the host virtualizes guest operating system real memory addresses and transparently shares the system's real memory pages among the guest virtual machines in which each of the operating systems execute.
Specifically, the guest operating systems use virtualized real memory for the QDIO queues. However, the adapters only access host real memory locations and not the virtualized guest real memory locations. This is because the adapter does not have a dynamic address translation mechanism, like the central processing units. Thus, the virtualized real memory used for QDIO queues by a guest operating system is transparently mapped by the host into different memory pages in actual or physical memory. Therefore, the hypervisor must perform various actions in order to allow guest operating systems to use QDIO adapters.
First, it must mirror the QDIO queues created by the guest operating system in guest real memory, by creating shadow copies of the guest QDIO queues in host real memory. These shadow queues are transparently created and maintained by the host, and are in turn, implicitly used by the guest to access the QDIO adapters. Only the shadow queues are connected to a QDIO adapter and only the host real memory I/O data area addresses contained in the QDIO shadow queues are actually used to transfer data to/from the adapter and main memory.
Further, the host must keep each guest created QDIO queue in synchronism with its corresponding host created shadow QDIO queue that is used by the QDIO adapter to perform the guest operating system specified QDIO operations. In particular, all of the QDIO state information in the guest QDIO queues in guest real memory must be kept in synchronism with the state information in the corresponding host shadow QDIO queues in host real memory; and the operating system specified guest real memory I/O data area addresses in the guest QDIO queues must be assigned corresponding host real memory data areas by the host and these data area addresses must then be specified in the host created shadow queues used by the adapter.
This process of shadowing and maintaining synchronization requires that the host take control of the central processor each time a guest operating system executes a QDIO operation. Further, the processing cycles associated with the transfer of control from/to the guest operating system and the host each time the guest performs a QDIO operation significantly increases the total processor overhead required for QDIO processing over that which is necessary when the operating system is executing in a native machine environment. This additional processing overhead can contribute to as much as a 40% increase in the total processing cycles required to run a workload as a pageable guest, as compared to executing it natively.
Based on the foregoing, a need exists for a capability that enhances input/output processing. For example, a need exists for a capability that reduces the host processing overhead associated with performing I/O operations, such as QDIO operations.
The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a computer program product for facilitating data transfer in a pageable mode virtual environment. The computer program product includes, for instance, a computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method including obtaining by a processor of the pageable mode virtual environment an indication of a buffer to be used in data transfer, the buffer being associated with a guest program operating in a pageable mode virtual machine of the pageable mode virtual environment; and performing by a processor of the pageable mode virtual environment processing associated with the buffer to enable data transfer, the performing being absent intervention of a host of the pageable mode virtual environment, wherein the performing comprises changing a state of the buffer between program owned and adapter owned.
In a further embodiment, a computer program product for facilitating processing of input/output (I/O) operations in a pageable mode virtual environment is provided. The computer program product includes, for instance, a computer readable storage medium readable by a processor and storing instructions for execution by the processor for performing a method including processing an I/O operation of a guest program operating in a virtual machine of the pageable mode virtual environment, wherein the processing is absent intervention by a host of the pageable mode virtual environment.
Methods and systems corresponding to the above-summarized computer program products are also described and may be claimed herein, as well as other aspects of the invention.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
a depicts one embodiment of a computing environment to incorporate and use one or more aspects of the present invention;
b depicts further details of an I/O subsystem of
a-2b depict one embodiment of the logic associated with guest configuration, in accordance with an aspect of the present invention;
a depicts one example of a request block for a Store Channel Subsystem Characteristics command, in accordance with an aspect of the present invention;
b depicts one example of a response block for the Store Channel Subsystem Characteristics command, in accordance with an aspect of the present invention;
a depicts one example of a request block for a Store Subchannel QDIO Data Command, in accordance with an aspect of the present invention;
b depicts one example of a response block for the Store Subchannel QDIO Data Command, in accordance with an aspect of the present invention;
c depicts one example of a subchannel QDIO description block of the response block of
a depicts one example of a QDIO Buffer Information Control Block (QBICB), in accordance with an aspect of the present invention;
b depicts one example of a queue storage descriptor of the QBICB of
c depicts one example of a Fibre Channel Protocol (FCP) operation block (FOB), in accordance with an aspect of the present invention;
a depicts one example of a request block for a Set Channel Subsystem Characteristics command, in accordance with an aspect of the present invention;
b depicts one example of an operation data area of the request block of
c depicts one example of a response block for the Set Channel Subsystem Characteristics command, in accordance with an aspect of the present invention;
a-8c depict one embodiment of the logic associated with a Set QDIO Buffer State instruction for use with network protocol I/O requests, in accordance with an aspect of the present invention;
a depicts one example of a page status table entry (PGSTE), in accordance with an aspect of the present invention;
b depicts one example of a page table entry (PTE), in accordance with an aspect of the present invention;
a-13b depict one embodiment of the logic associated with a resolve function, in accordance with an aspect of the present invention;
a-14c depict one embodiment of the logic associated with an Extract QDIO Buffer State instruction for use with network protocol I/O requests, in accordance with an aspect of the present invention;
a-18d depict one embodiment of the logic associated with a Set QDIO Buffer State instruction for use with fibre channel protocol I/O requests; and
a-19e depict one embodiment of the logic associated with an Extract QDIO Buffer State instruction for use with fibre channel protocol I/O requests.
In accordance with an aspect of the present invention, I/O operation requests from pageable storage guests (e.g., V=V guests) are interpreted without host intervention. For example, a capability is provided which allows a host, such as a z/VM® hypervisor, and hardware/firmware of a processor, such as a zSeries processor, to interact with each other in a controlled cooperative manner in order to process V=V guest operating system QDIO operations without requiring the transfer of control from/to the guest operating system and the host.
Further, in another aspect of the present invention, Host Page Management Assist functions are provided that can be invoked during the interpretation of a pageable storage mode guest. These functions include a resolve host page function used to dynamically resolve a host page invalid condition; a pin host page function used to indicate that the contents of a host page are pinned in a host page frame; and an unpin host page by virtual address function and/or an unpin host page by PTE address function used to indicate that the contents of a host page are no longer pinned in a host page frame.
One embodiment of a computing environment to incorporate and use one or more aspects of the present invention is described with reference to
As one example, computing environment 100 includes a central processor complex (CPC) 102 providing virtual machine support. CPC 102 includes, for instance, one or more virtual machines 104, one or more central processors 106, at least one host 108 (e.g., a control program, such as a hypervisor), and an input/output subsystem 110, each of which is described below.
The virtual machine support of the CPC provides the ability to operate large numbers of virtual machines, each capable of hosting a guest operating system 112, such as Linux. Each virtual machine 104 is capable of functioning as a separate system. That is, each virtual machine can be independently reset, host a guest operating system, and operate with different programs. An operating system or application program running in a virtual machine appears to have access to a full and complete system, but in reality, only a portion of it is available.
In this particular example, the model of virtual machines is a V=V model, in which the memory of a virtual machine is backed by virtual memory, instead of real memory. Each virtual machine has a virtual linear memory space. The physical resources are owned by host 108, and the shared physical resources are dispatched by the host to the guest operating systems, as needed, to meet their processing demands. This V=V virtual machine model assumes that the interactions between the guest operating systems and the physical shared machine resources are controlled by the host, since the large number of guests typically precludes the host from simply partitioning and assigning the hardware resources to the configured guests. One or more aspects of a V=V model are further described in an IBM publication entitled “z/VM: Running Guest Operating Systems,” IBM Publication No. SC24-5997-02, October 2001, which is hereby incorporated herein by reference in its entirety.
Central processors 106 are physical processor resources that are assignable to a virtual machine. For instance, virtual machine 104 includes one or more logical processors, each of which represents all or a share of a physical processor resource 106 that may by dynamically allocated to the virtual machine. Virtual machines 104 are managed by host 108. As examples, the host may be implemented in microcode running on processors 106 or be part of a host operating system executing on the machine. In one example, host 108 is a VM hypervisor, such as z/VM®, offered by International Business Machines Corporation, Armonk, N.Y. One embodiment of z/VM® is described in an IBM publication entitled “z/VM: General Information Manual,” IBM Publication No. GC24-5991-04, October 2001, which is hereby incorporated herein by reference in its entirety.
Input/output subsystem 110 directs the flow of information between devices and main storage. It is coupled to the central processing complex, in that it can be part of the central processing complex or separate therefrom. The I/O subsystem relieves the central processors of the task of communicating directly with the I/O devices coupled to the CPC and permits data processing to proceed concurrently with I/O processing. In one embodiment, the I/O subsystem is based on the Queued Direct I/O (QDIO) architecture, one example of which is described in a United States Patent entitled “Method And Apparatus For Providing Configuration Information Using A Queued Direct Input-Output Device,” Markos et al., U.S. Pat. No. 6,519,645 B2, issued Feb. 11, 2003, which is hereby incorporated herein by reference in its entirety. Further details regarding I/O subsystem 110 are described with reference to
I/O subsystem 110 includes a plurality of adapters 120 coupled to a plurality of I/O devices 122. A particular adapter may be coupled to one or more I/O devices and an I/O device may be coupled to one or more adapters.
An example of an adapter is a QDIO adapter. A QDIO adapter provides the functions typically associated with an input/output control unit as well as the functions typically associated with an input/output channel. A QDIO adapter has the ability to directly access main storage for the purpose of exchanging data with a program that has access to the same main storage.
A logical representation of the QDIO adapter is a QDIO control unit. There may be more than one QDIO control unit representing the same QDIO adapter.
The QDIO adapter is connected to other elements in the CPC via a channel path that provides a data transmission path between the adapter and main storage. To transmit data received from an attached I/O device or network of devices to a program, the adapter uses a QDIO input queue located within main storage. Similarly, to transmit data to a QDIO adapter, which in turn may transmit data to an attached I/O device or network of devices, a program uses a QDIO output queue.
There may be a plurality of (e.g., 32) input queues and output queues available to a particular adapter. To indicate which queues are available to a particular adapter, a logical structure referred to as a QDIO subchannel is employed. The QDIO subchannel includes zero or more input queues and zero or more output queues, and one or more subchannels are associated with a particular adapter giving the adapter access to those queues.
For both QDIO input queues and QDIO output queues, main storage (e.g., host storage) is used as the medium by which data is exchanged between the program the adapter. Additionally, these queues provide the ability for both the program and the adapter to directly communicate with each other in an asynchronous manner that is both predictable and efficient without requiring the services of a centralized controlling mechanism, such as an operating system input/output supervisor, and the resulting overhead such a control mechanism implies. Both input and output queues are constructed in main storage by the program and are established and activated at the QDIO adapter. These queues are used in communications during processing of I/O operations.
In accordance with an aspect of the present invention, the host (e.g., z/VM) and processor (e.g., zSeries) hardware/firmware interact with each other in a controlled cooperative manner in order to process V=V guest operating system QDIO operations without requiring transfer of control from/to the guest operating system and the host. Guest QDIO operations may take advantage of a facility called QDIO Enhanced Buffer State Management (QEBSM), and when this is done, the operations can directly pass through the host to/from the QDIO adapter without host intervention via a facility referred to as the QDIO Enhanced Buffer State Management Interpretation (QEBSMI) facility that allows selected instructions to be interpretatively executed for a pageable storage mode guest. One example of interpretive execution (e.g., Start Interpretive Execution (SIE)) is described in System/370 Extended Architecture/Interpretative Execution, IBM Publication No. SA22-7095-01, September 1985, which is hereby incorporated herein by reference in its entirety.
In order to configure a system to be able to use QEBSM and QEBSMI various configuration tasks are performed. As described with reference to
In one embodiment, guest program 200 (
The Store Channel Subsystem Characteristics command is used to store information about the characteristics of the I/O subsystem. The execution of the Store Channel Subsystem Characteristics command does not change any information contained in the I/O subsystem. The Store Channel Subsystem Characteristics command is executed synchronously and is not interpretively executed.
One embodiment of a request block for a Store Channel Subsystem Characteristics command is described with reference to
One embodiment of a response block for a Store Channel Subsystem Characteristics command is described with reference to
Returning to
Next, a determination is made as to the QDIO characteristics for the specified subchannel number, STEP 216. In one example, this determination is made via a CHSC Store Subchannel QDIO Data command issued by the program to the host. The host then issues the instruction to the real subchannel in order to acquire the queue characteristics, STEP 218. The machine processes the instruction, STEP 220.
The Store Subchannel QDIO Data command is used to obtain self-description information for the QDIO adapters associated with a specified range of subchannels.
Execution of the Store Subchannel QDIO Data command does not change any information contained in the I/O subsystem. In one example, it is executed synchronously.
A command request block 400 (
One embodiment of a response block for the Store Subchannel QDIO Data command is described with reference to
At least one subchannel QDIO description block is stored, and the actual number of blocks stored is determined by subtracting eight bytes from the size of the command response block and dividing the remainder by 32.
Subchannel QDIO description blocks are stored for some or all of the specified subchannels, beginning with the subchannel specified by the first subchannel number field. Subchannel QDIO description blocks are stored in ascending order of subchannel numbers.
One example of a subchannel QDIO description block is described with reference to
Adapter description information that is stored in the subchannel QDIO description block is current at the time the Store Subchannel QDIO Data command is executed. However, some of the adapter description information may change as a result of the successful completion of an Establish QDIO Queues CCW command, described below, by the specified adapter. Thus, the Store Subchannel QDIO Data command is to be reissued after successful completion of an Establish QDIO Queues CCW command to obtain the most current description for the following:
The remaining adapter description information is current regardless of successful completion of an Establish QDIO Queues command.
Referring back to
Employing the information returned in CHSC response data, QDIO queues are created by the program in main storage, STEP 228. A QDIO queue includes data structures that describe the queue as well as buffer storage blocks that are used for data transfer. In one example, the multiple separate data structures, called queue components, that collectively describe the queue's characteristics and provide the controls to allow the exchange of data between the program and the adapter include, for instance:
Further details regarding QDIO queues are provided in U.S. Pat. No. 6,519,645, entitled “Method And Apparatus For Providing Configuration Information Using A Queued Direct Input-Output Device,” Markos et al., issued Feb. 11, 2003, and U.S. Pat. No. 6,345,241, entitled “Method And Apparatus For Simulation Of Data In A Virtual Environment Using A Queued Direct Input-Output Device,” Brice et al., issued Feb. 5, 2002, each of which is hereby incorporated herein by reference in its entirety.
Returning to
Thereafter, the QDIO queues are established in the adapter, STEP 232. As one example, this is accomplished by the program issuing a Start Subchannel instruction with an Establish QDIO Queues Channel Command Word, an example of the processing of which is described in U.S. Pat. No. 6,332,171, entitled “Self-Contained Queues With Associated Control Information For Receipt And Transfer Of Incoming And Outgoing Data Using A Queued Direct Input-Output Device,” Baskey et al., issued Dec. 18, 2001, which is hereby incorporated herein by reference in its entirety.
The Establish QDIO Queues Command is used to write a queue descriptor record (QDR) to the I/O device. The control unit (e.g., adapter) associated with the I/O device uses the information in the queue descriptor record to locate the QDIO control blocks that are used to establish prioritized data queues. These queues are subsequently used for the direct transmission of data to and from the I/O device.
The establish procedure causes the host to create a set of shadow QDIO queues in host storage for the queues being established by the program, STEP 234. That is, the host builds shadow versions of some of the QDIO control structures, including, for instance, the storage list and the storage block address list. Also, the SLSB is allocated in host storage, although it contains no addresses, because of its definition as the controlling mechanism for the programs and the adapters cooperating processes. The queue descriptor record associated with the Establish QDIO Queues command includes the main memory addresses of the QDIO queue components as seen by the program. The host translates those addresses, as well as addresses within the SL and SBALs, in building its own copy of the queue descriptor record and the shadow SL and SBALs. Further details regarding shadowing and processing associated with the QDR are included in the aforementioned United States patents that have been incorporated herein by reference in their entirety.
Subsequent to creating the shadow queues, if the QEBSM enablement bit is set in the QIB, then the host creates a structure, referred to as the QDIO Buffer Information Control Block (QBICB) in host storage to map both the guest's and the host's shadow QDIO queues to the machine, STEP 236. The QBICB includes pointers and information describing the guest and host structures. It is associated with the QDIO subchannel, so that when that subchannel is identified as an operand of SQBS and EQBS, described below, the guest and host structures can be located and used. Further details regarding the QBICB are described with reference to
A QDIO Buffer Information Control Block (QBICB) 600 (
In one example, QBICB 600 includes the following fields:
FOBs may be linked to one another to form various lists. One list is the available FOB list. When the FOB has been allocated by the host and is not being used for a QDIO operation, it resides in the available FOB list (AFOBL). In one instance, a separate available FOB list is provided for each processor. The FOBs in the available FOB list are singly linked using the next request head FOB address field. The last entry in the list is designated by a zero value in its next request head FOB address field. The origin of the available FOB list and the count of FOBs in the available FOB list are designated by fields in a CPU prefix area.
Another list is the request component FOB list that includes one or more FOBs that comprise a single, active FCP request. The number of FOBs in a request component FOB list corresponds to the number of SBALs in the FCP request. The first FOB in the request component FOB list is the head FOB and is used to represent the presence of the active request in the active request head FOB list.
For a single SBAL (i.e., single buffer) FCP request, the request component list is comprised of the head FOB. After the head FOB is constructed, it is placed at the beginning of the active request head FOB list and construction complete and request active indicators in the FOB's flags field are both set to one to indicate that the construction of the total FCP request is complete and the request is active.
For a multiple SBAL (i.e., multi-buffer) FCP request, one FOB is constructed corresponding to each SBAL that comprises the request. The first FOB constructed becomes the head FOB. The head FOB is added to the beginning of the active request head FOB list and, after it is constructed, the request active bit in the FOB's flags field is set to one to indicate that the request is active. The anchor of the list of the remaining request component FOBs resides in the head FOB.
The remaining FOBs are added to the request component FOB list and are singly linked using the next request component FOB address field, beginning with the head FOB. The last FOB in the list is designated by a zero value in its next request component FOB address field. After the last FOB has been added to the list, the construction complete indicator in the head FOB's flags field is set to one to indicate that the construction of the total FCP request is complete and the request is active.
The FOBs in the request component FOB list including the head FOB are considered request component FOBs.
Another list is the active request head FOB list. When an FCP request is active, the request's head FOB is in the active request head FOB list. The active request head FOB list is a doubly linked list using the next request head FOB address and the previous request head FOB address fields. The anchors of the active request head FOB list are the first active request head FOB address and the last active request head FOB address fields in the queue storage descriptor for the queue. The first entry in the list is designated by a value of the address of the queue storage descriptor's first active request head FOB address field in the head FOB's previous request head FOB address field and the value of the address of the first entry's head FOB in the first active request head FOB address of the queue storage descriptor. The last entry of the list is designated by a value of the address of the queue storage descriptor's first active request head FOB address field in the head FOB next request head FOB address field and the value of the address of the last entry's head FOB in the last active request head FOB address of the queue descriptor.
Returning to
Thereafter, the program retrieves the subchannel token for the QEBSM connection and the type of SIGAs required (or desired) to manage this connection, STEP 250. In one example, this is performed by issuing the CHSC Store Subchannel QDIO Data command described above.
The host associates the QBICB with the subchannel, STEP 252. In one example, this association is performed by issuing a Set Channel Subsystem Characteristics command. The CHSC Set Channel Subsystem Characteristics command sets selected channel subsystem controls based on the operation code of a command request block for this command.
One example of a command request block 700 is described with reference to
As one example, the operation data area of the set adapter device controls operation includes, for instance, the following fields described with reference to
One example of a command response block 740 for the Set Channel Subsystem Characteristics command is described with reference to
Returning to
Moreover, the host retrieves the latest QDIO characteristics requested by the program, STEP 256. In one example, this is performed using the Store Subchannel QDIO Data command described herein. The Store Subchannel QDIO Data instruction is processed by the machine, STEP 258, and information is passed back to the host.
The host indicates that QEBSM is enabled and returns the host absolute address of the QBICB as the subchannel token and the SIGAs required in the program's CHSC response data, STEP 260. The program receives the subchannel token and characteristics for the connection, STEP 262.
Thereafter, the program initializes the QDIO buffers to the appropriate start-up states using an SQBS instruction with the returned subchannel token, STEP 264. The SQBS instruction, which is described below, is processed by the machine, STEP 266, and information is passed back to the program.
Moreover, a Start Subchannel with an Activate QDIO Queues Channel Command Word is issued by the program to activate the queues for data transfer, STEP 268. One example of the processing associated with this command is described in U.S. Pat. No. 6,332,171, entitled “Self-Contained Queues With Associated Control Information For Receipt And Transfer Of Incoming And Outgoing Data Using A Queued Direct Input-Output Device,” Baskey et al., issued Dec. 18, 2001, which is hereby incorporated herein by reference in its entirety. The host receives this Start Subchannel command and issues a Start Subchannel with an Activate QDIO Queues Channel Command Word informing the adapter to activate the queues for data transfer, STEP 270. Then the machine sends the Activate QDIO Queues to the adapter, STEP 272. When the request is initiated, QDIO data transfer may begin, STEP 274.
As part of QDIO data transfer, the guest program manages the QDIO buffer states. To do this, two instructions have been implemented, in accordance with an aspect of the present invention. One instruction is referred to as the Set QDIO Buffer State (SQBS) instruction and the other is referred to as the Extract QDIO Buffer State (EQBS) instruction. These instructions allow for dynamically synchronizing V=V virtual machine (guest) QDIO queues with the host shadow copies of those queues. The dynamic queue synchronization is performed by the machine during guest execution without host intervention. Thus, use of the Set QDIO Buffer State and Extract QDIO Buffer State instructions by the guest eliminates the context switching overhead that has been previously incurred in order to keep guest QDIO queues and host shadow QDIO queues synchronized.
One embodiment of using the Set QDIO Buffer State instruction is described with reference to
The processed frame descriptor list (PFDL) is a list of frame descriptors that describes host page frames that have been used to resolve host page invalid conditions during guest interpretation. The host page frames that are described by the PFDL have been assigned to host pages that provide storage for a guest. The PFDL is designated by a PFDL origin (PFDLO) at a specified host real address. The contents of the PFDLO, with five zeros appended on the right, specify the host home space virtual address of the first frame descriptor on the PFDL. A value of zero indicates that the list is empty.
The PFDLO is initialized by the host and may be changed by the host or a host page management assist function (described herein). The PFDLO is changed, in one embodiment, by means of a doubleword concurrent interlocked update operation that maintains the integrity of the list.
A frame descriptor describes a host page frame, and multiple frame descriptors may be linked to one another to form a list, such as the PFDL or an available frame descriptor list (AFDL). The frame descriptor is allocated, deallocated, and initialized by the host and may be updated by host page management assist functions (described herein). In one example, a frame descriptor 900 (
Returning to
The contents of the AFDLO, with five zeros appended on the right, specify the host home space virtual address of the first frame descriptor on the AFDL. A value of zero indicates that the list is empty.
The AFDLO is initialized by the host and may be changed by the host or host page management assist functions. The AFDLO is changed by means of a non-interlocked update operation.
A frame descriptor exists in one of the two lists: The processed frame descriptor list (PFDL) or the available frame descriptor list (AFDL). A separate pair of these lists is provided for each CPU. The origins of the AFDL and PFDL for a CPU are designated by means of fields in the prefix area of the CPU.
Thereafter, a host can start interpretive execution (SIE) of the program, STEP 804. In response to starting interpretive execution, the program begins to execute, and during execution, selects the next QDIO buffer to be transferred to the adapter, STEP 806. The program fills in the SBAL with the program absolute addresses of the storage blocks to be used in the data transfer, STEP 808. Further, the program transfers ownership of the buffer to the adapter by issuing a Set QDIO Buffer State instruction changing the state of the QDIO buffer from program owned to adapter owned, STEP 810. One example of the format of the Set QDIO Buffer State instruction is described with reference to
With the Set QDIO Buffer State instruction, the state of one or more QDIO input queue or output queue buffers, specified by the general registers designated by the R1 and R3 fields, are set to the state specified by the second operand. The operation proceeds until the specified buffer states are set or until a CPU determined number of buffer states have been set, whichever occurs first. The result is indicated in the condition code.
General register 1 includes the subchannel token that designates the QDIO subchannel that is enabled for QDIO Enhanced Buffer State Management (QEBSM) and is to have one or more of its buffer states set. (The subchannel token is obtained via the CHSC Store Subchannel QDIO Data command after the QDIO subchannel is enabled for QEBSM.)
The general register designated by the R1 field includes a specification of the QDIO queue and the first buffer whose state is to be set. When the instruction is issued, the general register designated by the R1 field includes the following fields:
The general register designated by the R3 field includes a count of the number of buffers whose states are set. The R3 register also includes a condition code qualifier field that is set along with the setting of the condition code when the instruction completes. When the instruction is issued, the general register designed by the R3 field includes the following fields:
The buffer number space is in the range of 0 to 127, as one example. The count value, in conjunction with the R1 BNUM value, may specify a wrap-around in the buffer number space.
The second operand address is not used to address data. Instead, the rightmost eight bits of the second operand address specify the QDIO buffer state to be set in the form of a storage list state block (SLSB) buffer state indicator. If the second operand address specifies a state that is neither an adapter owned nor a program owned state or the second operand address specifies a state that is both an adapter owned state and a program owned state (i.e., if bit 0 and bit 1 of the specified state are equal), a condition code is set and the R3 condition code qualifier is set with a prespecified value.
The unit of operation for the Set QDIO Buffer State includes setting one buffer state, decrementing the count field in the R3 register by one, and incrementing the buffer number field in the R1 register by one. If incrementing the buffer number field causes the buffer number to exceed the maximum buffer number of 127, a wrap-around in the buffer number space occurs and the buffer number is reset to 0.
Instruction operation completes when any one of the following occurs first:
A serialization and checkpoint synchronization function is performed before and after the specified buffer states are set.
The execution of the Set QDIO Buffer State instruction does not change any information contained in the subchannel nor does it cause the I/O subsystem to signal the adapter nor does it alter any buffer state information in program storage.
The processing associated with executing the SQBS instruction is described with reference to
Referring to
Next, the machine performs a subchannel determination, STEP 818. In one example, the SQBS subchannel token is used to locate the QBICB which contains the SID identifying the subchannel. A determination is made as to whether the subchannel token matches the token stored in the subchannel specified by the QBICB. If not, then processing is completed by the host, STEP 820. Otherwise, processing continues with the machine.
In further detail, when a Set QDIO Buffer State instruction is executed (as well as a Signal Adapter instruction or an Extract QDIO Buffer State instruction), a subchannel token in general register 1 is used to designate a subchannel. This token is used to determine whether the designated subchannel is enabled for QDIO Enhanced Buffer State Management Interpretation and to determine the subsystem id of the subchannel. In one example, the steps used to make these determinations include the following:
Next, a determination is made by the machine as to whether the SQBS operands are valid, STEP 821. As examples, the buffer number is validated. In one example, if the R1 buffer number field specifies a value greater than 127, the instruction completes, and a condition code and condition code qualifier are set. Additionally, the count value is validated. If the R3 count field specifies a value of 0 or a value greater than 128, the instruction once again completes, and a condition code and condition code qualifier are set. Additionally, the specified buffer state is checked for validity. If the buffer state specified by the second operand address specifies a simultaneous adapter owned and program owned state or a state that is neither adapter owned or program owned, the instruction completes, and a condition code and condition code qualifier are set. Further, the queue storage descriptor for the specified queue is located. If the R1 queue index field is greater than or equal to the value in the QBICB queue count field, the specified queue is not established for the subchannel and the instruction completes, and a condition code and condition code qualifier are set. Otherwise, the value of the R1 queue index field is used as the index into the QBICB queue descriptor address array to determine the queue descriptor address.
Should any of the above tests fails, then processing completes, and a condition code and condition code qualifier are set, STEP 822. Otherwise, processing continues.
An interval completion indicator in the guest state is set to one and a serialization and checkpoint synchronization function is performed. In case of a machine failure, this indicator informs the host that the host control structures may be in a questionable state.
Next, an attempt is made to acquire the queue storage descriptor lock for the specified queue, STEP 824. The queue storage descriptor address is fetched from the queue storage address field in the QBICB queue descriptor for the specified queue. An attempt is made, using an interlocked update operation, to change the QSD lockword value from available to locked by the machine. If the QSD address is zero or an access exception condition exists when attempting to set the lockword, a validity interception is recognized and a validity interception reason code is set.
If the interlocked update operation fails, this operation may be retried a model dependent number of times. If the QSD lockword cannot be set to the locked by machine value, a partial execution interception is recognized, the partial execution reason code in the QEBSM-PEI parameters field returned to the host program is set to, for instance, 1, and processing is completed by the host, STEP 826. If the interlocked update operation succeeds, processing continues.
The following logic is repeated for the number of QDIO buffers specified by the count field on the SQBS instruction. Initially, the SQBN and SBAL for the QDIO buffer to be processed are located, STEP 828. In one example, the host storage list for the specified queue is located. That is, the host storage list address is fetched from the host storage list address field in the queue storage descriptor. Then, the specified queue storage list state block is located. This address is fetched from the SLSB address field in the queue storage descriptor. The first SLSB entry (SQBN) to be set is located. The value of the R1 buffer number (BNUM) field is used as the index into the SLSB to determine the address of the first SQBN that is to be set. If an access exception condition exists when accessing the SQBN, a validity interception is recognized and a validity interception reason code is set.
Thereafter, a determination is made as to whether the specified buffer is currently owned by the program, STEP 830. In one example, this determination is made by checking the SQBN. If not, then processing completes, and a condition code and condition code qualifier are set, STEP 832. Otherwise, processing continues with the machine making a determination as to whether the state of the specified buffer needs to be extracted, STEP 834 (
Further, for each meaningful host SBALE that has a valid storage block, the following functions are performed, STEP 842: (1) The program absolute (host virtual) storage block address is fetched from the host SBALE, STEP 844; (2) An HPMA pin function, described below, is invoked to translate and pin the page in storage, STEP 846. If the HPMA pin function is unsuccessful, STEP 848, then processing is completed by the host, STEP 850. Otherwise, processing continues by the machine; (4) The host absolute address of the storage block returned by the HPMA pin is moved into the host's SBALE, STEP 852; and (5) The pinned page indicator in the host's SBAL is set, STEP 854 (
Thereafter, the buffer's SQBN is set to the adapter owned state specified by the SQBS instruction, STEP 856. In one example, the state is specified by the second operand. Additionally, the SQBS buffer number (BNUM) is incremented by one, STEP 858, and the SQBS count field is decremented by one, STEP 860.
A determination is made as to whether the SQBS count is equal to 0, STEP 862. Should the SQBS count not be equal to 0, then processing continues with locate the SQBN and SBAL for the QDIO buffer to be processed, STEP 828 (
If this is the last QDIO buffer to transfer at this time, then an appropriate SIGA instruction function is issued to signal the adapter that an I/O request is pending, for instance, SIGA-wt for an output operation or SIGA-rt when making an input available for subsequent use by the adapter, STEP 868.
It is noted that if the SQBS processing had to be completed by the host, then the host presents the appropriate end ops to the program, instead of the machine.
In describing the SQBS instruction, reference is made to Host Page Management Assist (HPMA). In accordance with an aspect of the present invention, Host Page Management Assist includes four assist functions that can be invoked by other CPU facilities during the interpretation of a pageable storage guest. These functions include a pin function, a resolve host page function, an unpin host page by virtual address function and an unpin host page by PTE address function, each of which is described below.
The HPMA pin function invoked from the SQBS instruction provides the CPU with the ability to indicate that the contents of a host page are pinned in a host page frame. A page may be pinned when it is required to be resident beyond the execution of a single instruction (e.g., for the duration of an I/O operation). When a page is in the pinned state, this is an indication to the host that the page is ineligible for page invalidation. When a page is successfully pinned by the pin host page function, a pinned page counter associated with the page may be incremented.
In one example, a delta pinned page counter array (DPPCA) is an array of, for instance, 31-bit signed integer pinned page counters, one of which may be incremented by the pin host page function when a page is successfully pinned. Further, one of the DPPCA counters may be decremented by an unpin host page by virtual address function or an unpin host page by PTE address function (described below), when a page is successfully unpinned. In one implementation, one counter in the DPPCA is associated with pages that reside in host real storage below a given size (e.g., 2 GB), and the other counter is associated with pages that reside in host real storage at or above that size.
The host classifies each page as either a class 0 page or class 1 page and sets a page class indicator in a page status table entry (PGSTE) for the page accordingly. One example of a page status table entry 1100 is described with reference to
A PGSTE corresponds to a page table entry (PTE), an example of which is described with reference to
Further details regarding page table entries and page tables, as well as segment table entries mentioned herein, are provided in an IBM® publication entitled, “z/Architecture Principles of Operation,” IBM® Publication No. SA22-7832-02, June 2003, which is hereby incorporated herein by reference in its entirety.
Continuing with the discussion of DPPCA, a separate DPPCA is associated with each virtual configuration for class 0 pin page counts and is designated by the class 0 DPPCA origin field in each description of the virtual configuration. A single DPPCA exists for class 1 pin page counts and is designated by the class 1 DPPCA origin field in the prefix area.
A DPPCA is not subject to key controlled protection nor low address protection. In one example, it is an 8-byte control block that is allocated on a doubleword boundary, and includes, for instance, a delta pinned page count for page frame real addresses less than a certain size (e.g., 2 GB), and a delta pinned page count for page frame real addresses greater than or equal to that size. An interlocked update operation is to be used to change this field in order to maintain the accuracy of its content. A delta pinned page counter may be incremented when a page is successfully pinned by the pin host page function.
When the pin host page function is invoked, the following data is input to the function: the host virtual address (page) to be pinned; the effective host address space control element (ASCE) that identifies the address space assigned to the guest and the translation tables to be used; an indication of whether the page is being pinned for a fetch or store operation; and a page content replacement indicator.
One embodiment of the logic associated with the pin host page function is described with reference to
The located PTE/PGSTE pair is locked using, for instance, an interlocked update operation to change the page control lock indicator in the PGSTE from 0 to 1, STEP 1202. If the pair could not be locked or if the page control interlock (PCL) was already one, INQUIRY 1204, then a failure is indicated, STEP 1206. Otherwise, processing continues with determining whether the pin count for the page would overflow with the next pin (i.e., if the value of the pin count field in the PGSTE is at a predefined value (e.g., 255)), INQUIRY 1208. If so, then processing ends with a failure, STEP 1206. Otherwise, the PTE for the virtual page to be pinned is fetched, STEP 1210.
Next, a determination is made as to whether the page is protected and is being pinned for a store operation, INQUIRY 1212. That is, if the input indication is for a store operation and the page protection bit is one in either a segment table entry or page table entry for the page, then failure is indicated, STEP 1206. Otherwise, processing continues with determining whether the page is invalid, INQUIRY 1214. If the page is invalid, then a resolve host page function is invoked, described below, STEP 1216. If the resolve host page function fails, INQUIRY 1217, then the pin function fails, STEP 1206. However, if the resolve is successful, or if the page is not invalid, then the pin count is incremented, STEP 1218.
Subsequently, a determination is made as to whether the pin state is being changed from the unpinned to the pinned state, INQUIRY 1220. If so, the associated delta pinned page count (i.e., the one selected based on page class and PFRA) is incremented by one, STEP 1222. That is, if the pin count is now one and the pin count is not in an overflow state (PGSTE.PCL=0), then the page state has transitioned from the unpinned to the pinned state. In this case, the delta pinned page count is incremented by one. Thereafter, or if the page state is not being changed from the unpinned to the pinned state, then the PTE/PGSTE pair is unlocked, STEP 1224. For example, the page control interlock bit in the PGSTE is set to zero, and processing is complete with a success.
When the pin host page function exits with an indication of success, the designated page is pinned and also may have been resolved by invocation of the resolve host page function. In this case, the page count for the page has been incremented in the status table by the machine and host translation tables may also have been updated by the machine and the previous requirement to transfer control to the host for page pinning and resolution is alleviated.
When the page host pin function exits with an indication of failure, no permanent change to the translation and status tables have been made, and any page controlled interlock bit that was set by the pin function is reset when the failure is recognized. In this case, the machine behaves as if the function was not invoked and the host will be given control to handle the failure.
As described above, if the page to be pinned is invalid, then a resolve host page function is invoked. The resolve host page function makes use of a host provided list of available host page frames that can be used by the CPU, when conditions permit, to dynamically resolve a host page invalid condition. The function also provides a means by which the host is informed about any host page frames that have been used to resolve host page invalid conditions by being assigned to specific host pages.
If host page management assist is enabled, the resolve host page function is invoked for a host page, and the page invalid bit in the page table entry (PTE.I) that describes the page is one, then one of the host page frames that the host has made available may be used to dynamically resolve the host page invalid condition.
Input to the resolve host page function includes the available frame descriptor list, the virtual address (page) to be resolved, the effective address space control element (ASCE) that identifies the address space and the translation tables to be used, a page content replacement indicator, indication of whether an interval completion indicator has already been set by the invoker, and an indication of whether access to the PTE/PTSTE pair has been locked by the invoker.
One embodiment of the logic associated with the resolve function is described with reference to
However, if there are available frames, then processing continues with locating the PTE/PGSTE pair for this page, STEP 1304. In one example, the ASCE and the virtual address (page) to be resolved are used to determine the location of the page table entry and page status table entry in host storage for the virtual address to be resolved.
Thereafter, the page frame real address is fetched from the first frame descriptor in the available frame descriptor list, STEP 1306. Also, if not already indicated as having been established by the invoker, the interval completion indicator is marked to show that the resolve host page function has begun and serialization and checkpoint synchronization functions are also performed, STEP 1308.
Subsequently, the PTE/PGSTE pair are conditionally locked, STEP 1310. If the resolve host page function entry conditions indicate that access to the PTE and PGSTE corresponding to the host virtual address to be resolved has not already been serialized, an interlocked update operation is used in an attempt to change the page control interlock indicator in the PGSTE (PGSTE.PCL) from zero to one. If the PTE/PGSTE pair cannot be locked or if the PGSTE.PCL is found to be already one, INQUIRY 1312, then processing exits with a failure, STEP 1302. Otherwise, processing continues with fetching the PTE for the virtual page to be resolved, STEP 1314. In one example, the page table entry is block-fetched concurrently from host storage.
Moreover, a determination is made as to whether the page needs to be resolved, INQUIRY 1316. In one example, this determination is made by checking the invalid indicator in the PTE. If the PTE indicates that the page is valid (PTE.I=0), then the page is resolved and processing is complete. However, if the page is to be resolved, then a further determination is made as to whether the page can be resolved, INQUIRY 1318. For instance, if the PGSTE for the virtual page to be resolved indicates that the page contents are not logically zero (PGSTE.Z=0) and the input page content replacement indicator indicates the page cannot be replaced, then processing exits with a failure, STEP 1302. However, if the page content logically zero indicator in the PGSTE is one or the resolve host page function entry conditions indicate that the content of the host virtual page to be resolved can be replaced, then an attempt to resolve the page can be made and processing continues, STEP 1320.
To resolve the page, various steps are performed as described with reference to
Additionally, the frame descriptor is removed from the available frame descriptor list, STEP 1332. As one example, a non-interlocked update operation is used to remove the frame descriptor from the available frame descriptor list. The contents of the next frame descriptor address field of the frame descriptor replace the old value in the AFDLO.
Next, a frame assignment is recorded, STEP 1334. The PTE for the host virtual page is preserved by copying the PTE into the PTE copy field of the frame descriptor. Additionally, the address of the PTE for the host virtual page is stored into the page table entry address field of the frame descriptor.
Next, a determination is made as to whether the page contents can be replaced, INQUIRY 1338. As one example, this determination is made by checking the input page content replacement indicator. If the input page content replacement indicator indicates that page contents can be replaced, the page content replacement indicator in the PGSTE (PGSTE.PCR) is set to one, STEP 1340. This allows the host to keep statistics on the number of operations that permit page content replacement.
Thereafter, or if the contents cannot be replaced, the PTE for the host virtual page is updated to remove the page invalid condition. For example, a concurrent store operation replaces the contents of the page frame real address field of the page table entry with the contents of the page frame real address field of the frame descriptor, and sets the page invalid bit in the page table entry to zero, STEP 1342. The other bits in the PTE remain unchanged. Moreover, the frame descriptor is added to the processed list, STEP 1344. This is performed via an interlocked update operation. This completes processing of resolve.
Returning to
Moreover, if the interval completion indicator was set by this function, serialization and checkpoint synchronization functions are performed and the interval completion indicator is reset, STEP 1324. This completes processing of the resolve function.
When the resolve function exits with an indication of success, the designated page has been resolved. In this case, the host translation tables have been updated by the machine and the previous requirement to transfer control to the host for page resolution is alleviated.
When the resolve function exits with an indication of failure, no changes to the translation tables have been made. In this case, the machine behaves as if the function was not invoked and the host is given control to handle the failure.
In addition to the SQBS instruction, the EQBS instruction has been implemented, in accordance with an aspect of the present invention. As examples, the guest issues an Extract QDIO Buffer State instruction when the guest receives notice from the adapter or on its own initiative when the buffer state is to be extracted. One embodiment of the logic associated with the Extract QDIO Buffer State instruction is described with reference to
Prior to running the program, certain initialization takes place. For example, the host performs various cleanup tasks, STEP 1400 (
During execution, the program selects the next outstanding QDIO buffer the program is waiting for the adapter to finish processing, STEP 1406. The program then issues the EQBS instruction to retrieve the current state of the selected buffers, STEP 1408.
One example of the format of an EQBS instruction is described with reference to
With the Extract QDIO Buffer State instruction, the state of one or more same state QDIO input queue or output queue buffers, specified by the general registers designated by the R1 and R3 fields, is extracted and loaded in the form of a QDIO buffer state indicator into the general register designated by the R2 field. Optionally, specified buffers that are in the input buffer primed state may be changed to the input buffer acknowledged state. The operation proceeds until the specified buffers have their common state extracted or until at least some of the specified buffers have had their common state extracted and the next buffer state to be extracted is different from those previously extracted or until a CPU determined number of buffer states have had their common state extracted, whichever comes first. The result is indicated in the condition code.
General register 1 includes the subchannel token that designates the QDIO subchannel that is enabled for QDIO Enhanced Buffer State Management (QEBSM) and is to have one or more of its buffer states extracted.
The general register designated by the R1 field includes a specification of the QDIO queue and the first buffer whose state is to be extracted. When the instruction is issued, the general register designated by the R1 field includes the following fields:
The general register designed by the R3 field includes a count of the number of buffers whose states are to be extracted. The R3 register also includes a condition code qualifier that is set along with the setting of the condition code when the instruction completes. When the instruction is issued, the general register designated by the R3 field includes the following fields, in one example:
Before the instruction is issued, the general register designated by the R2 field includes an indication of whether specified buffer states to be extracted and found to be in the input buffer primed state should be changed to the input buffer acknowledged state after the buffer state is extracted. When the instruction completes, the resulting condition code and condition code qualifier indicate, for instance, whether the register designated by the R2 field includes the state of one or more buffers. The first buffer to which this state applies is identified by the values of the R1 QIX and BNUM fields when the instruction is issued. The count of sequentially numbered buffers to which this state applies is the difference between the R3 count field when the instruction is issued and the decremented R3 count field when the instruction completes.
The general register designated by the R2 field includes, for instance, the following fields:
The unit of operation for the Extract QDIO Buffer State includes extracting one buffer state, loading the state into the R2 BSTATE field, optionally changing the buffer state, decrementing the count field in the R3 register by one, and incrementing the buffer number field in the R1 register by one. If incrementing the buffer number field causes the buffer number to exceed the maximum buffer number of, for instance, 127, a wrap around in the buffer number space occurs and the buffer number is reset to, for instance, 0.
Instruction operation completes when one of the following occurs first:
A serialization and checkpoint synchronization function is performed before and after the specified buffer states are extracted.
The execution of Extract QDIO Buffer State does not change any information contained in the subchannel nor does it cause the I/O subsystem to interrogate or signal the adapter.
The processing associated with executing the EQBS instruction is described with reference to
Referring to
Next, the machine performs a subchannel determination, STEP 1414. In one example, the subchannel token in general register 1 is used to locate the QBICB which includes the SID identifying the subchannel. A determination is made as to whether the subchannel token matches a token stored in the subchannel specified by the QBICB. If not, then processing is completed by the host; otherwise, processing continues with the machine.
A determination is made as to whether the EQBS operands are valid, STEP 1416. In one example, the buffer number is validated. If the R1 buffer number (BNUM) field specifies a value greater than, for instance, 127, the instruction completes, and a condition code and condition code qualifier are set, STEP 1418. Further, the count value is validated. If the R3 count field specifies a value of zero or a value greater than, for instance, 128, the instruction completes, and a condition code and condition code qualifier are set. Further, the queue storage descriptor for the specified queue is located. If the R1 queue index field is greater than or equal to the value in the QBICB queue count field, the specified queue is not established for the subchannel, the instruction completes, and a condition code and condition code qualifier are set. Otherwise, the value of the R1 queue index field is used as the index into the QBICB queue descriptor address array to determine the queue descriptor address.
Thereafter, an attempt is made to acquire by the machine the queue storage descriptor lock for the specified queue, STEP 1420. The QSD address is fetched from the queue storage address field in the QBICB queue descriptor for the specified queue. An attempt is made, using an interlocked update operation, to change the QSD lockword value from available to locked by the machine. If the QSD address is 0 or an access exception condition exists when attempting to set the lockword, an exception is recognized. If the interlocked update operation fails, this operation may be retried a model-dependent number of times. If the QSD lockword cannot be set to the predefined value, a partial execution interception is recognized and processing continues with the host, STEP 1422. However, if the interlocked update operation succeeds, processing continues with the machine.
Next, the SQBN for the first QDIO buffer is located, STEP 1424 (
The following logic is repeated for the number of QDIO buffers specified by the count field on the EQBS instruction. Initially, the SQBN and SBAL for the QDIO buffer to be processed are located, STEP 1426. Next, a determination is made as to whether this SQBN matches the STATE to be returned, STEP 1428. That is, if at least one buffer state has been extracted and the SQBN for the current buffer includes a designation of a state that is different from that previously extracted, the unit of operation is suppressed, the EQBS instruction completes, and a condition code and condition code qualifier are set, STEP 1430. Otherwise, processing continues.
If the specified buffer is currently owned by the program and needs to be extracted (buffer synchronization in host SBAL=1), STEP 1431, then the following is performed for all SBALEs (e.g., 16) in the SBAL:
If the acknowledged input buffer control is on and the value in the SQBN indicates input buffer primed, then the SQBN is set to the input buffer acknowledged state, STEP 1446. The EQBS buffer number is also incremented by one, STEP 1448, and the EQBS count field is decremented by one, STEP 1450.
Next, a determination is made as to whether the EQBS count is equal to zero, STEP 1452. If not, then processing continues with locate the SQBN and SBAL for the QDIO buffer to be processed, at STEP 1426 (
If the EQBS processing had to be completed by the host, then the host presents the appropriate condition code and condition code qualifier to the program, instead of the machine.
As described above, the EQBS instruction may invoke the HPMA unpin by virtual address function. This function provides the CPU with the ability to indicate that the contents of a host page no longer need to be pinned in a host page frame on behalf of the I/O request. The page to be unpinned is specified by a host virtual address. One or more pinned page counters that are associated with a page may be decremented when a page is successfully unpinned.
In one example, the inputs to this function include the virtual address (page) to be unpinned; the effective address space control element (ASCE) that identifies the address space assigned to the guest and the translation tables to be used; and the host real address of the page to be unpinned.
To facilitate host data structure validity determination, the invoker of the unpin host page by virtual address function is responsible for ensuring that the interval completion indicator is set throughout the checkpoint interval in which the function is invoked. The unpin host page by virtual address function does not examine or modify the interval competition indicator.
One embodiment of the logic associated with the unpin host page by virtual address function is described with reference to
A determination is made as to whether the pin count for the page will underflow with the next unpin (i.e., if the value of the pin count field in the PGSTE is zero), INQUIRY 1608. If so, then processing once again completes with a failure, STEP 1606; otherwise, processing continues with fetching the PTE of the virtual page to be unpinned, STEP 1610. In one example, the page table entry is fetched block-concurrently from host storage.
If the page is invalid or the page invalid bit is one, INQUIRY 1612, then processing once again ends in a failure, STEP 1606. Otherwise, a determination is made as to whether an incorrect PTE has been located, INQUIRY 1614. That is, if the page frame real address field in the PTE does not designate the same page as the input host real address of the page to be unpinned, then an incorrect PTE has been located, and processing ends, STEP 1606. Otherwise, processing continues. The pin count field in the PGSTE is decremented by one, STEP 1616, and a determination is made as to whether the page state is being changed from pinned to unpinned, INQUIRY 1618. If the page state is being changed from the pinned to the unpinned state, then the associated delta pinned page count is decremented by one, STEP 1620. That is, if the pin count is now zero and the pin count is not in an overflow state (PGSTE.PCO=0), then the page state has transitioned from the pinned to the unpinned state, and the delta pinned page count is decremented by one.
Thereafter, or if the page state has not changed from pinned to unpinned, then the PTE/PGSTE pair is unlocked by, for instance, setting the page control interlock indicator to zero, STEP 1622. This completes processing of the unpin by virtual address function.
When the unpin function exits with an indication of failure, no changes to the status tables have been made. In this case, the machine behaves as if the function was not invoked and the host is given control to handle the failure. When this function exits with an indication of success, the designated page has been unpinned. In this case, the host status tables have been updated by the machine and the previous requirement to transfer control to the host for page pinning is alleviated.
Another function that can be invoked is an unpin host page by PTE address function. This function provides a CPU with the ability to indicate that the contents of a host page are no longer pinned in the host page frame. The page to be unpinned is specified by the host real or absolute address of the host page table entry. One or more pinned page counters that are associated with a page table may be decremented when the page is successfully unpinned.
In one example, the input to the unpin host page by PTE address function includes the host real address or the host page table entry for the page to be unpinned.
One embodiment of the logic associated with the unpin host page by PTE address function is described with reference to
On the other hand, if the lock was successful, then a further determination is made as to whether the pin counts of the page will underflow with the next unpin (i.e., if the value of the pin count field in the PGSTE is zero), INQUIRY 1708. If so, then processing once again exits in failure, STEP 1706. Otherwise, the PTE for the virtual page to be unpinned is fetched, STEP 1710. In one example, it is fetched block-concurrently from host storage.
Thereafter, a determination is made as to whether the page is invalid, INQUIRY 1712. In particular, the page invalid indicator is checked. If it is one, then the page is invalid and processing exits in failure, STEP 1706. If the page is valid, then the pin count field in the PGSTE is decremented by one, STEP 1714.
Next, a determination is made as to whether the page state is being changed from pinned to unpinned, INQUIRY 1716. If the pin count is now zero and the pin count is not in an overflow state, then the page state has transitioned from the pinned to the unpinned state. Thus, the delta pinned page count is decremented by one, STEP 1718. Thereafter, or if the page state is not being changed from pinned to unpinned, then the PTE/PGSTE pair is unlocked, STEP 1720. In one example, this includes setting the page control interlock indicator to zero. This concludes processing of the unpin page by PTE address function.
When the unpin function exits with an indication of failure, no changes to the status tables have been made. In this case, the machine behaves as if the function was not invoked and the host is given control to handle the failure. When this function exits with an indication of success, the designated page has been unpinned. In this case, the host status tables have been updated by the machine and the previous requirement to transfer control to the host for page pinning is alleviated.
Described in detail above is processing associated with interpretating I/O operation requests from pageable storage guests absent host intervention. The requests can be requests to network protocol (e.g., internet protocol) or network devices via QDIO adapters or similar adapters based on a network protocol. As a further example, the requests can be requests to fibre channel attached devices based on a fibre channel protocol (FCP).
An FCP adapter is connected to other elements in the CPC via a channel path between the adapter and main storage, similar to non-FCP adapters, such as QDIO adapters. However, processing associated with FCP adapters is different in some respects than processing associated with non-FCP adapters. For instance, data input and output is performed via a QDIO output queue and the QDIO input queue is used only for the adapter to signal completion of an I/O operation. Further, when the FCP adapter places an output queue buffer in the program owned state, the I/O operation, including the transfer of data, may not be totally complete. (This is contrary to the QDIO architecture for non-FCP adapters.) Even though the buffer is in the program owned state, the program is not to perform any action on the buffer pages until the adapter signals completion of the I/O operation through the input queue. However, the program may perform actions on the QDIO data structures associated with the buffers, as long as the buffer pages remain untouched.
When an FCP operation is initiated, a unique request id is passed from the program to the adapter. It is this request id that is returned in the input queue that signals that the operation is totally complete and that the program may interact with the buffer pages.
Thus, with FCP, there is the ability to disconnect the buffer pages from the QDIO data structures. In other words, copies of the addresses of the buffer pages in the QDIO data structures could be kept in another location and new buffer pages can be allocated, their addresses placed into the QDIO data structures, specifically into the desired SBALEs, and a new I/O operation initiated—all while waiting for the request id from the adapter that the operation has completed on the original set of buffer pages (whose addresses where copied).
To facilitate this retention of a description of the disconnected buffer pages, the FCP Operations Block (FOB) was designed. The FOB is a structure that is built when SQBS is issued by the guest specifying an FCP request, and as the machine resolves and pins pages, it records the host page table entry addresses of the buffer pages in the FOB. The machine also records the guest's request id in the FOB and replaces it with a request id that is equal to the address of the FOB. Then, later when the adapter returns that request id in the input queue and the guest issues EQBS to interrogate the state of the input queue, the machine can locate the FOB, unpin the pages, and replace the request id with the original guest value that was recorded earlier.
Further details regarding processing for FCP are described with reference to
Referring initially to
Subsequent to performing cleanup and/or ensuring there are sufficient frames on the relevant lists, a host can start interpretive execution of the program, STEP 1804. The program begins to execute and during execution, the program selects the next QDIO buffer to be transferred to the adapter, STEP 1806. The program fills in the SBAL with the program absolute addresses of the storage blocks to be used for the data transfer, STEP 1808. Further, the program transfers ownership of the buffer to the adapter by issuing a Set QDIO Buffer State instruction changing the state of the QDIO buffer from program owned to adapter owned, STEP 1810.
In response to the SQBS instruction being issued, the machine performs a host enablement check to determine whether the host enabled the guest for the QEBSM facility, STEP 1812. If a QEBSMI enablement control (e.g., in guest state) is set to zero indicating non-enablement, STEP 1814, then processing is to be completed by the host instead of the machine, STEP 1816. When a condition is encountered during Set QDIO Buffer State interpretation that requires some action or service from the host, either an instruction interception or a partial execution interception (PEI) is recognized, an interception parameter is set with the Set QDIO Buffer State instruction text, and for PEI, QEBSM-PEI parameters are set in guest state.
Next, the machine performs a subchannel determination, STEP 1818, in which a determination is made as to whether the subchannel token matches the token stored in the subchannel specified by the QBICB. If not, then processing is completed by the host, STEP 1820. Otherwise, processing continues with the machine.
Moreover, a determination is made by the machine as to whether the SQBS operands are valid, STEP 1822. If any of the operands are invalid, then processing ends, a condition code and condition code qualifier are set, STEP 1824. Otherwise, processing continues.
Next, an attempt is made to acquire the queue storage descriptor lock for the specified queue, STEP 1826. If the QSD lock is not acquired, then processing is completed by the host, STEP 1828. Otherwise, processing continues by the machine.
The following logic is repeated for the number of QDIO buffers specified by the count field in the SQBS instruction. Initially, the SQBN, program SBAL and corresponding host SBAL for the QDIO buffer to be processed are located, STEP 1830. Thereafter, a determination is made as to whether the specified buffer is currently owned by the program, STEP 1832. If the buffer is not owned by the program, then processing completes with a condition code and condition code qualifier, STEP 1834. Otherwise, processing continues with a determination as to whether the state of the specified buffer is to be extracted with the EQBS instruction, STEP 1836 (
Further, if the SBAL buffer sequence bits indicate this is a new FCP request (as opposed to the continuation of a request spanning multiple SBALs), then the previous FCP request, if any, is marked as completed, STEP 1844. For instance, the host SBALE 0 entry is checked for an indication of a new request: If the buffer sequence type in the SBAL-Flags (SBALF) field of the guest SBALE 0 indicates that a command is specified, the pending request component FOB list completion bit in the queue flags field of the QSD address is one, and there are FOBs in the active request head FOB list, the construction complete bit in the flags field of the first head FOB in the list is set to 1, and the pending request component FOB list complete bit in the queue flags field of the QSD address is set to zero.
Next, a determination is made as to whether there is an FOB on the available FOB list, STEP 1846. If not, then processing is completed by the host, STEP 1848. Otherwise, processing continues by the machine. For example, the first available FOB from the available FOB list is removed from that list and the available FOB count is decremented by one, STEP 1850. Then, the new FOB is added to the appropriate list based on the buffer sequence bits in the SBAL, STEP 1852. For instance, if this is the first or only SBAL, then this is a new FCP request. In that case, the FOB is a head FOB and is inserted on the active request FOB list and the request is marked active in the FOB. However, if this is the middle or last SBAL, then this is a continuation or the last SBAL of a multiple SBAL FCP request. Thus, this FOB is a non-head component FOB and is added to the request component FOB list of the first head FOB on the active request FOB list anchored in the queue storage descriptor.
For a head FOB, the following processing occurs, STEP 1854:
Further, for each meaningful host SBALE that has a valid storage block address, the following functions are performed, STEP 1868:
If this is the last SBAL of a multiple SBAL or the only SBAL of an FCP request, then the request is marked as complete in the FOB and the pending request FOB list completion indicator is set to 0, STEP 1882.
Thereafter, the buffer's SQBN is set to the adapter owned state specified by the SQBS instruction, STEP 1884. Additionally, the SQBS buffer number is incremented by one, STEP 1886, and the SQBS count field is decremented by one, STEP 1888.
A determination is made as to whether the SQBS count is equal to zero, STEP 1890. Should the SQBS count not be equal to zero, then processing continues with locate the SQBN, program SBAL and host SBAL for the QDIO buffer to be processed, STEP 1830 (
In addition to the above changes associated with processing an SQBS instruction for the FCP protocol, there are changes associated with the EQBS instruction. One example of the logic associated with the EQBS instruction for FCP requests is described with reference to
Prior to running the program, certain initialization takes place. For example, the host performs various cleanup tasks, STEP 1900, including, for instance, removing and processing frames on a processed frame descriptor list. Further, the host ensures that there are sufficient frames on the available frame descriptor list and the available FOB list, STEP 1902. Thereafter, the host starts interpretive execution of the program, STEP 1904.
During execution, the program selects the next outstanding QDIO buffer or buffers the program is waiting for the adapter to finish processing, STEP 1906. The program then issues the EQBS instruction to retrieve the current state of the selected buffer(s), STEP 1908.
In response to the EQBS instruction being issued, the machine performs a host enablement check to determine whether the host enabled the guest for the QEBSM facility, STEP 1910. If the guest is not a pageable storage mode guest or the QEBSMI facility enablement control is zero, INQUIRY 1912, then processing is to be completed by the host, STEP 1914. Otherwise, processing continues with the machine.
Next, the machine performs a subchannel determination, STEP 1916. If the subchannel token does not match the token stored in the subchannel specified by the QBICB, then processing is completed by the host, STEP 1914; otherwise, processing continues with the machine.
Further, a determination is made as to whether the EQBS operands are valid, STEP 1918. If one or more of the operands are invalid, then processing ends, and a condition code and condition code qualifier are set, STEP 1920. Otherwise, processing continues.
An attempt is made by the machine to acquire a queue storage descriptor lock for the specified queue, STEP 1922. If the QSD lock is not acquired, a partial execution interception is recognized and processing continues with the host, STEP 1924. However, if the lock is acquired (e.g., the interlocked update operation succeeds), processing continues with the machine.
Next, the SQBN for the first QDIO buffer is located, STEP 1926. The state of this buffer is the state that will be returned upon instruction completion in STATE.
The following logic is repeated for the number of QDIO buffers specified by the count field on the EQBS instruction. Initially, the SQBN, program SBAL, and corresponding host SBAL for the QDIO buffer to be processed are located, STEP 1928. Next, a determination is made as to whether the SQBN matches the STATE to be returned, STEP 1930 (
If the queue is an FCP (e.g., format 1) output queue and the SQBN specifies a program owned state, then the buffer synchronization required and EQBS operation required indicators in the host SBAL are turned off, STEP 1934. Moreover, the SBALF fields, except for SBALF 2, are copied into the corresponding fields in the program SBALE, STEP 1936.
If the specified buffer is currently owned by the program and it is a format 1 input queue, then the following steps are performed for meaningful SBALEs for which the SBALE request identifier field is not zero in the host SBAL, STEP 1938:
2) The head FOB returned by the adapter is validated for this user by verifying that the subchannel token in the QBICB specified in the FOB matches the token specified on the SQBS instruction. If the FOB is not for this user, STEP 1944, then processing is completed by the host, STEP 1946; otherwise, processing continues with the machine.
4) It is verified that the FOB is currently on the active request list and is complete. If the request active and construction complete indicators are on and the request id posted indicator is off in the FOB flag, STEP 1952, then processing is completed by the host, STEP 1954; otherwise, processing continues with the machine.
If the acknowledged input buffer control is one and the value in the SQBN indicates the input buffer primed state, then the SQBN is set to the input buffer acknowledged state, STEP 1974.
Thereafter, the EQBS buffer number is incremented by one, STEP 1976, and the EQBS count field is decremented by one, STEP 1978. If the EQBS count field is not equal to zero, STEP 1980 (
As described above, the QDIO architecture is extended such that pageable storage mode guests may experience near native QDIO performance. This V=V QDIO passthrough capability can approach the same levels of processor efficiency as would be derived natively in the absence of the virtual machine operating environment. This is accomplished by relocating the QDIO buffer state information from the pageable storage mode guests execution address space into host (e.g., VM host) storage and implementing instructions to allow the guest to manipulate the buffer states and manage host translations and shadow structures accordingly. Two instructions are introduced that give the program the capability to manage QDIO buffer states by queue number/buffer number designation, instead of by directly addressing the buffer states in program storage. This abstracted form of buffer state designation allows the interpreted versions of these instructions to be implemented in a manner that improves the performance of pageable mode virtual machine QDIO operations. It also allows reduction in SIGA overhead (fewer SIGAs). Moreover, the subchannel token representation allows interpretation of SIGA for subchannels assigned in the program, thus eliminating host involvement in SIGA handling in most cases.
A QDIO subchannel is enabled for QDIO Enhanced Buffer State Management, when an Establish QDIO Queues CCW command that specifies a request for QDIO Enhanced Buffer State Management enablement successfully completes.
A QDIO subchannel is disabled for QDIO Enhanced Buffer State Management, when an Establish QDIO Queues CCW command that does not specify a request for QDIO Enhanced Buffer State Management enablement successfully executes or an action or event occurs which causes disestablishment of the queues from the subchannel.
Bit 7 of the QDIO adapter characteristics 1 (QDIOAC1) field in the subchannel QDIO data block that is part of the CHSC Store Subchannel QDIO Data command response block indicates whether QDIO Enhanced Buffer State Management is enabled for a specific subchannel.
When a QDIO subchannel is enabled for QDIO Enhanced Buffer State Management, the subchannel is designated by its subchannel token in general register 1 for the following instructions:
The subchannel token is obtained from the subchannel token field of the subchannel QDIO data block that is part of the response to a CHSC Store Subchannel QDIO data command that is executed after the subchannel has been enabled for QEBSM. Thus, to effect QEBSM enablement and to obtain a subchannel token for a specific subchannel, the program executes the following sequence:
After a QDIO subchannel has been enabled for QEBSM, the Set QDIO Buffer State (SQBS) instruction is used to change the states of the queue buffers active on the subchannel. Timely use of the Extract QDIO Buffer State (EQBS) instruction to interrogate the states of the queue buffers active on the subchannel allows host pages to be unpinned in a manner that does not unduly stress the storage over-commitment that is associated with the management of a high number of pageable storage mode guests. Alternatively, if a program is not well behaved or the demand for host storage is high, the host can unpin any pinned buffer page in the program owned state without waiting for a guest to issue EQBS.
Once the state of a buffer has been changed from a program owned state to an adapter owned state by Set QDIO Buffer State, the state is to be interrogated by Extract QDIO Buffer State and found to have reverted to a program owned state before another SET QDIO BUFFER STATE is used again for that buffer. In summary, the guest prepares the data buffers and executes the Set QDIO Buffer State (SQBS) instruction. The SQBS instruction attempts to ensure all specified buffer pages are resident. If a buffer page has not been referenced, host page management assist functions are used to dynamically assign a new frame in the page table, pin the page and update the host shadow QDIO SBAL entry. If, however, a buffer page exists, is currently paged out, and is an input queue buffer page, host page management assist functions are used to dynamically assign a new frame in the page table, pin the page, and update the host shadow QDIO SBAL entry.
Adapter initiated state changes from adapter owned to program owned state are directly observable by the guest via the Extract QDIO Buffer State (EQBS) instruction which specifies buffers that the program had earlier placed in an adapter owned state and returns the current buffer state.
In addition to the above, a host page management assist facility (HPMA) can be invoked by other CPU facilities during the interpretation of a pageable storage mode guest. One or more aspects of the HPMA facility may be used with or without the QDIO architecture. HPMA includes, for instance, four HPMA assist functions, such as a resolve host page function, a pin host page function, an unpin host page by virtual address function and an unpin host page by PTE address function.
If a host page invalid condition exists on an instruction fetch or instruction operand access and the host page management assist is enabled, the resolve host page function may be invoked. The resolve host page function may be used during the translation process to dynamically resolve a host page invalid condition and prevent a page translation exception from being recognized. If the resolve host page function completes successfully, in one example, the unit of operation for which the host page invalid condition existed is nullified and retried. If the resolve host page function completes unsuccessfully, the original host page invalid condition is handled, as if the host page management assist is not enabled.
QEBSM in conjunction with HPMA provides the following advantages and performance improvement: Less SIE entry/exit overhead and less overhead for host context switching and routing (more cycles for productive work); eliminates host shadow copy of QDIO buffer states and associated synchronization overhead; dynamic updating of page status table and translation tables; dynamic prevention of host page faults; improved QDIO shadowing; and dynamic maintenance of host shadow structures by the CPU while in the interpretive execution mode without host intervention.
While various examples and embodiments are described herein, these are only examples, and many variations are included within the scope of the present invention. For example, the computing environment described herein is only one example. Many other environments, including other types of communications environments, may include one or more aspects of the present invention. For instance, different types of processors, guests and/or hosts may be employed. Moreover, other types of architectures can employ one or more aspects of the present invention.
Further, in the examples of the data structures and flows provided herein, the creation and/or use of different fields may include many variations, such as a different number of bits; bits in a different order; more, less or different bits than described herein; more, less or different fields; fields in a differing order; different sizes of fields; etc. Again, these fields were only provided as an example, and many variations may be included. Further, indicators and/or controls described herein may be of many different forms. For instance, they may be represented in a manner other than by bits. Additionally, although the term address is used herein, any designation may be used.
Moreover, although aspects of the invention are described with respect to an internet protocol and fibre channel protocol, again, these are only examples. Other protocols may benefit from one or more aspects of the present invention, including, but not limited to other network protocols.
Again, although HPMA is described with reference to QDIO and the examples described herein, one or more aspects of HPMA are usable by themselves. The use with QDIO is only one example.
As used herein, the term “page” is used to refer to a fixed-size or predefined-size area of storage. The size of the page can vary, although in the examples provided herein, a page is 4K. Similarly, a buffer is a structure to hold data and many types and sizes of structures are possible. Similarly, a storage block is a block of storage and as used herein, is equivalent to a page of storage. However, in other embodiments, there may be different sizes of blocks of storage and/or pages. Many other alternatives are possible. Further, although terms, such as queues, lists, tables, etc. are used herein, any types of data structures may be used. Again, those mentioned herein are just examples.
The present invention can be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has therein, for instance, computer readable program code means or logic (e.g., instructions, code, commands, etc.) to provide and facilitate the capabilities of the present invention. The article of manufacture can be included as a part of a computer system or sold separately.
Additionally, at least one program storage device readable by a machine embodying at least one program of instructions executable by the machine to perform the capabilities of the present invention can be provided.
The flow diagrams depicted herein are just examples. There may be many variations to these diagrams or the steps (or operations) described therein without departing from the spirit of the invention. For instance, the steps may be performed in a differing order, or steps may be added, deleted or modified. All of these variations are considered a part of the claimed invention.
Although preferred embodiments have been depicted and described in detail herein, it will be apparent to those skilled in the relevant art that various modifications, additions, substitutions and the like can be made without departing from the spirit of the invention and these are therefore considered to be within the scope of the invention as defined in the following claims.
This application claims priority to U.S. application Ser. No. 10/855,200, entitled “INTERPRETING I/O OPERATION REQUESTS FROM PAGEABLE GUESTS WITHOUT HOST INTERVENTION,” filed May 27, 2004, which is hereby incorporated herein by reference in its entirety. This application contains subject matter which is related to the subject matter of the following patents/applications, each of which is assigned to the same assignee as this application. Each of the below listed patents/applications is hereby incorporated herein by reference in its entirety: “SELF-CONTAINED QUEUES WITH ASSOCIATED CONTROL INFORMATION FOR RECEIPT AND TRANSFER OF INCOMING AND OUTGOING DATA USING A QUEUED DIRECT INPUT-OUTPUT DEVICE,” Baskey et al., U.S. Pat. No. 6,332,171 B1, issued Dec. 18, 2001; “METHOD AND APPARATUS FOR SIMULATION OF DATA IN A VIRTUAL ENVIRONMENT USING A QUEUED DIRECT INPUT-OUTPUT DEVICE,” Brice et al., U.S. Pat. No. 6,345,241 B1, issued Feb. 5, 2002; “METHOD AND APPARATUS FOR PROVIDING CONFIGURATION INFORMATION USING A QUEUED DIRECT INPUT-OUTPUT DEVICE,” Markos et al., U.S. Pat. No. 6,519,645 B2, issued Feb. 11, 2003; and “FACILITATING MANAGEMENT OF STORAGE OF A PAGEABLE MODE VIRTUAL ENVIRONMENT ABSENT INTERVENTION OF A HOST OF THE ENVIRONMENT,” Blandy et al., Ser. No. 10/854,990, filed May 27, 2004.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4516199 | Frieder et al. | May 1985 | A |
| 4564903 | Guyette et al. | Jan 1986 | A |
| 4649513 | Martin et al. | Mar 1987 | A |
| 4674038 | Brelsford et al. | Jun 1987 | A |
| 4785414 | Hemdal | Nov 1988 | A |
| 5109489 | Umeno et al. | Apr 1992 | A |
| 5185862 | Casper et al. | Feb 1993 | A |
| 5222215 | Chou et al. | Jun 1993 | A |
| 5253344 | Bostick et al. | Oct 1993 | A |
| 5276815 | Nakashima et al. | Jan 1994 | A |
| 5291605 | Takagi et al. | Mar 1994 | A |
| 5317705 | Gannon et al. | May 1994 | A |
| 5426748 | Brenza et al. | Jun 1995 | A |
| 5437033 | Inoue et al. | Jul 1995 | A |
| 5448702 | Garcia, Jr. et al. | Sep 1995 | A |
| 5459839 | Swarts et al. | Oct 1995 | A |
| 5499379 | Tanaka et al. | Mar 1996 | A |
| 5555385 | Osisek | Sep 1996 | A |
| 5555414 | Hough et al. | Sep 1996 | A |
| 5621912 | Borruso et al. | Apr 1997 | A |
| 5659794 | Caldarale et al. | Aug 1997 | A |
| 5687390 | McMillan, Jr. | Nov 1997 | A |
| 5765023 | Leger et al. | Jun 1998 | A |
| 5922057 | Holt | Jul 1999 | A |
| 5996026 | Onodera et al. | Nov 1999 | A |
| 6009275 | DeKoning et al. | Dec 1999 | A |
| 6014753 | Miyamoto et al. | Jan 2000 | A |
| 6021430 | Brandt et al. | Feb 2000 | A |
| 6023734 | Ratcliff et al. | Feb 2000 | A |
| 6044415 | Futral et al. | Mar 2000 | A |
| 6065089 | Hickerson et al. | May 2000 | A |
| 6081834 | Brandt et al. | Jun 2000 | A |
| 6085246 | Brandt et al. | Jul 2000 | A |
| 6105119 | Kerr et al. | Aug 2000 | A |
| 6141692 | Lowenstein et al. | Oct 2000 | A |
| 6237067 | Eberhard et al. | May 2001 | B1 |
| 6253224 | Brice, Jr. et al. | Jun 2001 | B1 |
| 6314501 | Gulick et al. | Nov 2001 | B1 |
| 6332171 | Baskey et al. | Dec 2001 | B1 |
| 6339801 | Hefferon et al. | Jan 2002 | B1 |
| 6345241 | Brice et al. | Feb 2002 | B1 |
| 6345327 | Baskey et al. | Feb 2002 | B1 |
| 6345329 | Baskey et al. | Feb 2002 | B1 |
| 6356979 | Sicola et al. | Mar 2002 | B1 |
| 6453403 | Czajkowski | Sep 2002 | B1 |
| 6496847 | Bugnion et al. | Dec 2002 | B1 |
| 6519645 | Markos et al. | Feb 2003 | B2 |
| 7114040 | Yoder | Sep 2006 | B2 |
| 7174429 | Revilla et al. | Feb 2007 | B2 |
| 7941799 | Easton et al. | May 2011 | B2 |
| 20050268071 | Blandy et al. | Dec 2005 | A1 |
| 20060294524 | Vega | Dec 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 0072107 | Feb 1983 | EP |
| 0419723 | Jan 1994 | EP |
| 58200363 | Nov 1983 | JP |
| 60150140 | Aug 1985 | JP |
| 62031437 | Feb 1987 | JP |
| 3037730 | Feb 1991 | JP |
| 9026889 | Jan 1997 | JP |
| 2001216171 | Aug 2001 | JP |
| WO2004102401 | Nov 2004 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20110173615 A1 | Jul 2011 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10855200 | May 2004 | US |
| Child | 13073362 | US |
