The present invention relates generally to data processing systems, and more particularly, to creation and utilization of a partition adjunct, comprising a new execution environment for the data processing system. The partition adjunct is utilized, in one aspect, as a dedicated adjunct providing a logical partition with access to a self-virtualizing input/output (I/O) device.
Logical partitions (LPARs) running atop a hypervisor of a data processing system are often used to provide higher-level function than provided by the hypervisor itself. For example, one LPAR may be designated a virtual input/output server (VIOS), which provides input/output services to one or more other LPARs of the data processing system. This offloading of higher-level function avoids complex code in the hypervisor, and thus, assists in maintaining the hypervisor small and secure within the data processing system. Using logical partitions to provide such services to other logical partitions, however, requires relatively high overhead to instantiate and run the logical partition, and thus, a full operating system, in order to provide such services.
Disclosed herein is a solution which allows services to be provided to a logical partition without requiring another full logical partition to provide the services, and without requiring embedding of the services in the initiating logical partition or in the hypervisor. A partition adjunct, such as described and claimed herein, is a new execution environment which provides this middle ground.
Provided herein, in one aspect, is a method of establishing access for multiple logical partitions to a self-virtualizing input/output device of a data processing system. The method includes: interfacing each logical partition of the multiple logical partitions to at least one associated partition adjunct instance, each partition adjunct instance coupling its associated logical partition of the multiple logical partitions to one of a virtual function or a queue pair of the self-virtualizing input/output device, each partition adjunct instance including a separate dispatchable state and being created employing virtual address space donated from the associated logical partition or a hypervisor of the data processing system, and each partition adjunct instance including a device driver for the virtual function or queue pair of the self-virtualizing input/output device; and providing each logical partition of the multiple logical partitions with at least one virtual input/output which is interfaced through the logical partition's at least one associated partition adjunct instance to the virtual function or queue pair of the self-virtualizing input/output device.
In another aspect, a data processing system is provided which includes: at least one processor comprising multiple logical partitions and multiple partition adjunct instances, which facilitate input/output access of the multiple logical partitions; and a self-virtualizing input/output device coupled to the multiple partition adjuncts. Each logical partition of the multiple logical partitions is interfaced to at least one associated partition adjunct instance. Each partition adjunct instance couples the associated logical partition of the multiple logical partitions to one of a virtual function or a queue pair of the self-virtualizing input/output device of the data processing system. Further, each partition adjunct instance includes a separate dispatchable state and employs virtual address space donated from the respective logical partition or a hypervisor of the data processing system. In addition, each partition adjunct instance comprises a device driver for a respective virtual function or queue pair of the self-virtualizing input/output adapter. Each logical partition of the multiple logical partitions comprises at least one virtual input/output which is interfaced through the logical partition's at least one associated partition adjunct instance to at least one virtual function or at least one queue pair of the self-virtualizing input/output device.
In a further aspect, the invention comprises an article of manufacture which includes at least one computer-readable medium having computer-readable program code logic to define a method of establishing access for multiple logical partitions to a self-virtualizing input/output device of a data processing system. The computer-readable program code logic, when executing on a processor, performing: interfacing of each logical partition of the multiple logical partitions to at least one associated partition adjunct instance, each partition adjunct instance coupling its associated logical partition of the multiple logical partitions to one of a virtual function or a queue pair of the self-virtualizing input/output device, and each partition adjunct instance comprising a separate dispatchable state and being created employing virtual address space donated from the associated logical partition or a hypervisor of the data processing system, and wherein each partition adjunct instance comprises a device driver for the virtual function or a queue pair of the self-virtualizing input/output device; and providing each logical partition of the multiple logical partitions with at least one virtual input/output which is interfaced through the logical partition's at least one associated partition adjunct instance to the virtual function or queue pair of the self-virtualizing input/output device.
Further, additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
SMP server computer system 100 includes a physical SMP server 102. Physical SMP server 102 includes physical hardware devices such as processor 104, memory 106, and I/O adapters 108. These physical devices are managed by hypervisor 110. Processors 104 are shared processors and each may be a simultaneous multithreading (SMT)-capable processor that is capable of concurrently executing multiple different threads on the processor.
A virtual server is a proxy for a physical server that has the same capabilities, interfaces, and state. Virtual servers are created and managed by a hypervisor that resides on physical SMP server computer system 100. A virtual server appears to be a physical SMP server to its user: the operating system, middleware, and application software that run upon it. SMP server computer system 100 includes one or more virtual servers such as virtual server 112 and virtual server 112a.
Each virtual server appears to its software to include its own processor(s), memory, and I/O adapter(s) that are available for the exclusive use of that virtual server. For example, virtual server 112 includes a virtual processor 120, virtual memory 122, and virtual I/O adapters 124. Virtual server 112a includes virtual processors 120a, virtual memory 122a, and virtual I/O adapters 124a.
Each virtual server supports its own software environment, including an operating system, middleware, and applications. The software environment of each virtual server can be different from the software environment of other virtual servers. For example, the operating systems executed by each virtual server may differ from one another.
For example, virtual server 112 supports operating system 114, middleware 116, and applications 118. Virtual server 112a supports operating system 114a, middleware 116a, and applications 118a. Operating systems 114 and 114a may be the same or different operating systems.
A virtual server is a logical description of a server that defines a server environment that acts, to a user, as if it were a physical server, being accessed and providing information in the same way as a physical server. The virtual processors, virtual memory, and virtual I/O adapters that are defined for each virtual server are logical substitutes for physical processors, memory, and I/O adapters.
Hypervisor 110 manages the mapping between the virtual servers with their virtual processors, virtual memory, and virtual I/O adapters and the physical hardware devices that are selected to implement these virtual devices. For example, when a virtual processor is dispatched, a physical processor, such as one of physical processors 104, is selected by hypervisor 110 to be used to execute and implement that virtual processor. Hypervisor 110 manages the selections of physical devices and their temporary assignment to virtual devices.
Hypervisor 110 services all of the logical partitions during a dispatch time slice. The dispatch time slice is a particular length of time. During each dispatch time slice, hypervisor 110 will allocate, or assign, the physical processor to each logical partition. When the logical partition has been allocated time on the physical processor, the virtual processors defined by that logical partition will be executed by the physical processor.
Hypervisor 110 is responsible for dynamically creating, managing, and destroying virtual SMP servers. Whole virtual processors, virtual I/O adapters, and virtual memory blocks can be removed or added by hypervisor 110. Hypervisor 110 is also responsible for dynamic resource allocation, managing time-sharing of physical resources, and altering the physical resource mapped to a processor without involving the operating system. Hypervisor 110 is also able to dedicate physical resources to virtual resources for situations where sharing is not desired. Hypervisor 110 is responsible for managing the addition or removal of physical resources. Hypervisor 110 makes these additions and deletions transparent to the upper level applications.
Also connected to system bus 206 is memory controller/cache 208, which provides an interface to local memory 209. I/O bus bridge 210 is connected to system bus 206 and provides an interface to I/O bus 212. Memory controller/cache 208 and I/O bus bridge 210 may be integrated as depicted.
Peripheral component interconnect (PCI) bus bridge 214 connected to I/O bus 212 provides an interface to PCI local bus 216. A number of modems may be connected to PCI bus 216. Typical PCI bus implementations will support four PCI expansion slots or add-in connectors. Communications links to network computers 108-112 in
Network adapter 220 includes a physical layer 282 which conditions analog signals to go out to the network, such as for example, an Ethernet network for an R45 connector. A media access controller (MAC) 280 is included within network adapter 220. Media access controller (MAC) 280 is coupled to bus 216 and processes digital network signals. MAC 280 serves as an interface between bus 216 and physical layer 282. MAC 280 performs a number of functions involved in the transmission and reception of data packets. For example, during the transmission of data, MAC 280 assembles the data to be transmitted into a packet with address and error detection fields. Conversely, during the reception of a packet, MAC 280 disassembles the packet and performs address checking and error detection. In addition, MAC 280 typically performs encoding/decoding of digital signals transmitted and performs preamble generation/removal as well as bit transmission/reception.
Additional PCI bus bridges 222 and 224 provide interfaces for additional PCI buses 226 and 228, from which additional modems or network adapters may be supported. In this manner, data processing system 200 allows connections to multiple network computers. A memory-mapped graphics adapter 230 and hard disk 232 may also be connected to I/O bus 212 as depicted, either directly or indirectly.
Service processor 204 interrogates system processors, memory components, and I/O bridges to generate and inventory and topology understanding of data processing system 200. Service processor 204 also executes Built-In-Self-Tests (BISTs), Basic Assurance Tests (BATs), and memory tests on all elements found by interrogating a system processor, memory controller, and I/O bridge. Any error information for failures detected during the BISTs, BATs, and memory tests are gathered and reported by service processor 204.
Those of ordinary skill in the art will appreciate that the hardware depicted in
The present invention may be executed within one of the computers or data processing systems depicted in
Disclosed hereinbelow is the concept of a partition adjunct, which is a partition that is more limited than a full, logical partition. As described below, the partition adjunct runs in a flat, static effective address space and problem state. These restrictions permit the hypervisor to apply a range of hypervisor and processor optimizations that result in a substantial decrease in system overhead associated with a context switch of the state machine from a logical partition to state data of a partition adjunct, that is, compared to a context switch of the state machine between logical partitions. In other respects, a partition adjunct is similar to a full logical partition. For example, a partition adjunct can be assigned resources, either physical or virtual, similar to a full logical partition. Further, a partition adjunct can be an end-point of a virtual input output (VIO) communications mechanism, similar to a full logical partition, such as a virtual input output server (VIOS).
Partition adjunct 320 is conceptually a child partition to client partition 310. The partition adjunct is less than a full logical partition, but is run in a manner whereby the hypervisor enforces security and isolation between the partition adjunct and the client partition it runs within. The partition adjunct is provided with reduced functionality compared with a full logical partition, for example, has no access to memory management unit (MMU) configuration or floating-point facilities, and is an environment that only the functionality needed to run the desired service (e.g., I/O driver) is provided. In the illustrated example, partition adjunct 320 includes a virtual I/O interface 322 and a hardware device driver service 324, which allows access to I/O device 350. In operation, client partition 310 accesses I/O device 350 via the partition adjunct 320, as illustrated. By reducing functionality within the partition adjunct environment, the run time overhead of dispatching and maintaining the partition adjunct (compared with another full logical partition) is reduced, and consequently, many of the performance disadvantages of using a separate logical partition as a virtual input output server (VIOS) are avoided.
As a specific example, the partition adjunct is described herein as running a reduced operating system environment for a device driver service. This service is provided by way of example only. The partition adjunct provides minimal, optimized, infrastructure comprising only (in one example) the structure needed by a device driver. For instance, if a Linux device driver is to run inside of the partition adjunct, then the minimal execution environment includes only the Linux kernel services or equivalent services, that the Linux device driver requires. If an AIX device driver is to run inside of the partition adjunct, then the minimal execution environment includes only the AIX kernel services, or equivalent services, that the AIX device driver requires. Advantageously, the partition adjunct runs in hypervisor/problem-state, directly against hypervisor interfaces. As explained in detail below, dispatching of the partition adjunct does not require a full partition context switch, which simplifies adjunct kernel requirements. This is achieved, in part, by mapping the partition adjunct into the client partition's virtual address page table. Client partition to partition adjunct isolation can be achieved, for example, via hypervisor-managed memory keys. Advantageously, the partition adjunct is not customer viewable. Further, the same partition adjunct service (referred to herein as a global partition adjunct service) may be instantiated within multiple client partitions, as explained below.
Unless otherwise specified, the partition adjunct embodiment described hereinbelow is assumed to be an execution environment that is part of a client partition, yet separated from the client partition by hypervisor isolation and security mechanisms, such as referenced below and described in greater detail in commonly assigned, co-filed application entitled “Hypervisor-Enforced Isolation of Entities Within a Single Logical Partition's Virtual Address Space”, Ser. No. 12/111,041, filed Apr. 28, 2008, and issued Aug. 30, 2011, as U.S. Pat. No. 8,010,763 B2, the entirety of which is hereby incorporated herein by reference. Further details of partition adjunct processing and usage are described in greater detail in commonly assigned, co-filed patent applications entitled: “Partition Adjunct for Data Processing System”, Ser. No. 12/110,923, filed Apr. 28, 2008, and published Feb. 5, 2009, as U.S. Patent Publication No. 2009/0037906 A1; “Client Partition Scheduling and Prioritization of Service Partition Work”, Ser. No. 12/110,953, filed Apr. 28, 2008, and published Feb. 5, 2009, as U.S. Patent Publication No. 2009/0037907 A1; and “Partition Adjunct With Non-Native Device Driver for Facilitating Access to a Physical Input/Output Device”, Ser. No. 12/110,982, filed Apr. 28, 2008, and published Feb. 5, 2009, as U.S. Patent Publication No. 2009/0037908 A1, the entirety of each of which is also hereby incorporated herein by reference.
In order for a partition adjunct to be a runable program, the hypervisor, along with a client partition that is to use the partition adjunct service, negotiate to establish the partition adjunct environment. Once this negotiation is complete, the client partition will have donated a portion of its virtual address space to the hypervisor for use by the partition adjunct. The hypervisor will use hardware and hypervisor facilities to ensure that the client partition no longer has access to or can modify the donated resources (e.g., the donated virtual address space). The hypervisor instantiates the effective address mappings required to run the partition adjunct using the donated virtual address resources. Subsequently, the hypervisor may switch between dispatching the client partition or the partition adjunct by reprogramming its control of the donated virtual address space. When the client partition runs, it may access all virtual address space assigned to it, except for the donated virtual address range, and when the partition adjunct runs, the hypervisor disables access to all virtual addresses of the client partition, except for the donated virtual address range, that is, the virtual address space to which it is enabled. This toggling of active/inactive virtual address ranges is significantly faster than reprogramming the full memory management and address translation hardware to effect a complete context switch of the current state machine between two full logical partitions, as is necessary to switch, for example, to a virtual input/output server partition. In this manner, the partition adjunct address space is carved out of and separated from the memory management and address translation hardware resources of the client partition. The partition adjunct is thus, from a processor's perspective, part of the client partition, but from the client partition's and hypervisor's perspective, is a distinct entity.
Advantageously, the partition adjunct concepts presented herein reduce the need to use full logical partitions for providing services to client partitions. This in turn frees up resources and improves performance for customer workloads. Additionally, the partition adjunct disclosed herein encourages the development and deployment of virtual platform services in lieu of development of operating system specific services by reducing the performance penalties associated with virtualized services. This in turn allows for savings and cost development, since services may be implemented only once (i.e., in a partition adjunct), rather than natively among multiple operating systems.
Various examples of protocol for instantiating and employing a partition adjunct are described below with reference to
Next, the hypervisor loads the partition adjunct program image into the donated adjunct memory 640 and initializes adjunct execution 645. Initializing a partition adjunct execution environment can include the hypervisor initializing effective address space to virtual address translation tables, adjunct protection keys, and special hypervisor problem state privileges. The hypervisor then dispatches the partition adjunct to begin execution at the partition adjunct program entry point 650, and the hypervisor sets up partition adjunct service interface calls for the client partition 655. The hypervisor calls or invokes the client partition per protocols or functions provided by the partition adjunct. The hypervisor then returns to the client partition with status of the partition adjunct instantiation 660.
Assuming that a partition adjunct image is available, then the client partition reserves or donates a portion of its virtual address space and memory space for the partition adjunct image 735, and invokes the hypervisor to instantiate the partition adjunct with the donated virtual address space and memory space, as well as the I/O device 740. The hypervisor creates the new partition adjunct and assigns the donated resources to the partition adjunct 745. The hypervisor then sets up appropriate hardware protection and enforcement of the donated virtual address space and memory space, so that the client partition can no longer access the donated space 750. A new dispatchable state for the partition adjunct is created by the hypervisor 755. As noted above, this new dispatchable state can advantageously be dispatched on a single hardware thread of an SMT data processing system, while other hardware threads continue to run the invoking (or requesting) client partition. The hypervisor loads the partition adjunct image into the donated adjunct memory 760, and initializes the partition adjunct execution environment 765. The partition adjunct is then dispatched to begin initialization execution at the partition adjunct image entry point 770. The hypervisor sets up partition adjunct service interface calls for the client partition, which include, for example, a virtual input/output request queuing interface for the subject virtualized device class 775. The hypervisor returns to the client partition with the status of the partition adjunct instantiation, and the new virtual device assigned to the client partition 780, and the client partition proceeds with normal virtual device driver load and configuration 785.
Assuming that the partition adjunct service is available, then the hypervisor receives the partition adjunct service request 1030, and determines whether to queue the service request or invoke the service request currently 1035. If the service request is to be queued, then the hypervisor queues the service request for the partition adjunct 1040, and returns to inquire whether the client partition needs to invoke a partition adjunct service 1005. If the service request is to be currently invoked, then the hypervisor performs a “light” context switch of the current state machine, and gives control of the processor to the partition adjunct 1050. This light context switch is explained in detail below with reference to
As illustrated in
In
In contrast,
As illustrated in
In
In one aspect, the above-described partition adjunct is utilized as a dedicated adjunct providing a logical partition with access to a physical input/output (I/O) device which is non-configurable by the logical partition. Currently, if a logical partition's operating system does not have access to a device driver for an assigned physical I/O adapter, the operating system may not directly use that I/O adapter. Obtaining function from the input/output adapter in such a case requires user intervention to reassign the device to, for example, a virtual input/output server (VIOS) partition, which includes creating the VIOS partition if it does not already exist, and creating virtual input/output channels between the logical partition and the VIOS.
In the examples of
Device driver development costs may be reduced if device drivers can be shared between operating systems. This is accomplished hereinbelow by deploying device drivers as partition adjunct services, in accordance with an aspect of the present invention. As noted above, partition adjuncts are lightweight execution environments which operate in a separate execution state from the conventional problem and privileged states of the user applications and kernel services, respectively. This new execution state is referred to herein as the hypervisor/problem state, and is illustrated in
Linux personality adjunct 2420 is conceptually a dedicated, child partition to AIX partition 2410. As described above, the partition adjunct is less than a full logical partition, but is running in a manner whereby the hypervisor enforces security and isolation between the partition adjunct and the AIX partition it runs with. The partition adjunct is provided with reduced functionality compared with a full logical partition. For example, the partition adjunct has no access to memory management unit (MMU) configuration or floating-point facilities, and is an environment wherein only the functionality needed to run the desired service (e.g., I/O driver) is provided.
In the illustrated example, the Linux personality adjunct 2420 includes a virtual I/O interface 2422 and a hardware device driver service 2424, which allows access to I/O device 2450. In this example, the hardware device driver service 2424 is a Linux hardware device driver, which runs within the Linux personality adjunct 2420 spawned by the AIX partition 2410 in response to the AIX partition noting that it had assigned to it an I/O device 2450 which was non-configurable by the AIX partition. The Linux personality adjunct 2420 includes non-native kernel services sufficient to run the Linux hardware device driver for the physical I/O device. These non-native kernel services are less than a corresponding, full operating system, that is, less than a full Linux operating system in this example.
In operation, AIX partition 2410 accesses I/O device 2450 via the virtual I/O interface 2416, 2422 between the AIX operating system 2414 and the Linux personality adjunct 2420, which includes the Linux hardware device driver 2424. By providing the non-native, Linux personality to the partition adjunct, the Linux hardware device driver is able to be run within the partition adjunct, and thereby provide access to an I/O device 2450 originally assigned to AIX partition 2410, notwithstanding that the I/O device is non-configurable by the AIX partition. The device becomes accessible to the AIX partition through the Linux personality adjunct 2420.
AIX personality adjunct 2420′ is again conceptually a child partition to client partition 2410′. The partition adjunct is less than a full logical partition, but is run in a manner whereby the hypervisor enforces security and isolation between the partition adjunct and the client partition it runs within, as described above. The partition adjunct is an environment wherein only the functionality needed to run the desired service (e.g., I/O driver) is provided. In this example, it is assumed that the adjunct is dedicated to the spawning logical partition, that is, Linux partition 2410′.
In the illustrated example, AIX personality adjunct 2420′ includes a virtual I/O interface 2422′ and the AIX hardware device driver 2424′, which allows access to I/O device 2450′. In operation, Linux partition 2410′ accesses I/O device 2450′ via the partition adjunct 2420′, as illustrated.
As used herein, a personality adjunct is a partition adjunct which has a particular operating system personality, but is less than the full operating system. In the implementation described herein, the personality adjunct is a non-native personality to the native operating system of the logical partition spawning the partition adjunct. For example, AIX partition 2410 of
By way of specific example, the Linux personality adjunct 2420 of
Protocol for creating the partition adjunct is discussed in greater detail below with reference to the exemplary flowchart of
Referring to
Determining whether a partition adjunct device driver image is available for a particular I/O device could be accomplished via look-up of registered partition adjunct images with the operating system, or by querying the platform firmware, or some other partition adjunct image server, to determine whether the partition adjunct device driver image exists and is capable of operating the assigned physical I/O device.
Assuming that a partition adjunct image is available for the I/O device, then the client partition reserves or donates a portion of its virtual address space and memory space for the partition adjunct image 735, and invokes the hypervisor to instantiate the partition adjunct with the donated virtual address space and memory space, as well as the I/O device 740. The operating system essentially dynamically gives up ownership of the physical I/O device, that it had no device driver for, by relinquishing ownership to the hypervisor. The hypervisor creates the new partition adjunct and assigns the donated resources to the partition adjunct 745. The hypervisor then sets up appropriate hardware protection and enforcement of the donated virtual address space and memory space, so that the client partition can no longer access the donated space 750. A new dispatchable state for the partition adjunct is created by the hypervisor 755.
As noted above, this new dispatchable state can advantageously be dispatched on a single hardware thread of an SMT data processing system, while other hardware threads continue to run the invoking (or requesting) client partition. The hypervisor loads the partition adjunct image into the donated adjunct memory 760, and initializes the partition adjunct execution environment 765. The partition adjunct is then dispatched to begin initialization execution at the partition adjunct image entry point 770. The hypervisor sets up partition adjunct service interface calls for the client partition, which include, for example, a virtual input/output request queuing interface for the subject virtualization device class 775. The hypervisor returns to the client partition with the status of the partition adjunct instantiation, and the new virtual I/O device assigned to the client partition 780. In this step, the hypervisor dynamically adds to the client partition a virtual I/O device, for which the client partition's native operating system has a device driver. The client partition then proceeds with normal virtual device driver load and configuration 785 for the virtual I/O device, which ultimately gains access to the physical I/O device that was non-configurable by the client partition.
To summarize, described hereinabove is a method of establishing access by a logical partition to a physical input/output (I/O) device owned by the logical partition. The method includes: creating a partition adjunct within the data processing system, the partition adjunct being created employing the physical I/O device and virtual address space donated from the logical partition to the partition adjunct, the logical partition including a native operating system, and the partition adjunct comprising a non-native operating system's device driver for the physical I/O device, the physical I/O device being non-configurable by the logical partition; and providing the logical partition with a virtual I/O device interfaced through the partition adjunct's non-native operating system's device driver to the donated physical I/O device. The creating is responsive, in one embodiment, to a logical partition determining that its operating system does not have a native device driver capable of configuring the physical I/O device assigned to it, and to the logical partition noting that a partition adjunct instance is available capable of running a non-native hardware device driver which supports the physical I/O device. In such a case, the hypervisor uses dynamic re-configuration facilities, as described above, to detach the physical input/output device from the logical partition and reassign the device to the partition adjunct being created. In return, the hypervisor creates a virtual device and assigns it to the logical partition. In this manner, the logical partition which started out owning a non-configurable physical input/output device ends up with a configurable, virtual I/O device, through which it can gain access to the physical I/O device via the partition adjunct. It is assumed that the operating systems at issue support such virtual I/O devices, since such facilities are required for operating systems to function in a multi-logical partition system where no physical adapters may have been assigned to the logical partition.
In another aspect, multiple partition adjuncts are employed to provide multiple logical partitions with access to, for example, a self-virtualizing input/output device, such as a self-virtualizing input/output adapter. In a virtualized system, if a single input/output adapter is present, and that adapter is to service multiple logical partitions of the data processing system, then either a single logical partition is permitted to multiplex the device (such as in a virtual input/output server approach illustrated in
Referring first to
Referring to
As an alternative, VIOS 2620 may be eliminated, with the physical function facility thereof moved to hypervisor 2630. Physical function 2650 within self-virtualizing input/output device 2640 provides the basic functionality need to configure virtual functions 2651, 2652, 2653. This physical function provides the core set of capabilities and defines enable bits for the virtual functions required. The native virtual function device drivers within the respective logical partitions are each native to the operating system of the respective logical partition. For example, LPAR A 2610A may comprise a Linux operating system, which requires a native Linux virtual function device driver, and LPAR B 2610B may comprise an AIX operating system, which employs a native AIX virtual function device driver 2615B.
In comparing the data processing system implementations of
One example of a self-virtualizing input/output device, such as employed in the data processing system illustrated in
Further details of self-virtualizing input/output devices, such as illustrated in
As an enhancement to the above-described data processing system alternatives of
In accordance with the enhancement, partition adjunct instances are deployed in a manner wherein each partition adjunct instance is created to support a particular logical partition to virtual function (or queue pair) pairing. Using this approach, each logical partition accesses a corresponding virtual function or queue pair employing abstract virtual input/output mechanisms, such as the VIO mechanisms employed in the VIOS implementation of
Another advantage of this partition adjunct implementation is that, since all partition adjunct instances are considered for the same device, they are able to share code and read only data (which is a feature described below in detail with reference to
Further, partition adjunct instances, since they are configured to support only one logical partition to virtual function (or queue pair) pairing at a time, may be readily written in a manner to avoid many of the synchronization and locking mechanisms required by traditional I/O stacks and drivers, both in the native device driver and the VIOS alternatives described above. For example, partition adjuncts may be written as polling state machines, and the dedicated nature of their runtime environment precludes the need to support active preemption, thus simplifying or eliminating the need for locking.
To summarize, the concepts described below permit logical partitions to obtain I/O services from input/output virtualization-capable, input/output devices or adapters in a manner than minimizes the device-driver development required for each operating system of the logical partition, since the operating system's only see virtual input/output (VIO) services (e.g., device driver services), not specific physical input/output adapter devices. This avoids the need to instantiate a logical partition to multiplex the underlying I/O hardware, and permits the multiplexing of I/O hardware to be accomplished via efficient VIO hardware capabilities, rather than software locks in a VIOS. This last aspect is a property that arises from the unique programming model of a partition adjunct, and assumes that a particular partition adjunct implementation used to support a particular I/O device makes use of these properties to create an efficient implementation.
Each virtual function is assigned to provide I/O services to a particular logical partition in the data processing system. The logical partitions in question avoid having direct interactions with their virtual functions, so as to maintain hardware abstraction, by employing partition adjunct instances instantiated for each logical partition to virtual function pairing. These partition adjuncts 2870A, 2871A, 2870B, 2870n are referred to as virtual function (VF) partition adjuncts due to the dedicated nature of the partition adjuncts to a particular logical partition to virtual function pairing. For all virtual functions associated with a same underlying hardware device, i.e., physical function 2850 or physical function 2880, the partition adjunct instances instantiated are the same. That is, each partition adjunct instance 2870A, 2870B . . . 2870n is the same, while partition adjunct instantiation 2871A is assumed to be different since it interfaces to a different virtual function 2881, associated with a different physical function 2880. Advantageously, by interjecting partition adjuncts between the logical partitions and the virtual functions, the input/output within each logical partition can be virtualized as a client virtual input/output (VIO) 2815A, 2815A′, 2815B . . . 2815n.
By way of specific example, the underlying physical function might comprise a peripheral component interconnect (PCI) function that supports the single root I/O virtualization capabilities (SR-IOV) defined in the above-referenced “Single Root I/O Virtualization and Sharing Specification”. The physical function would thus contain the SR-IOV capability structure. A virtual function is associated with the physical function, and shares one or more physical resources, such as a link, with the physical function and with other virtual functions that are associated with the same virtual function. A “function” means, in one embodiment, an addressable entity in configuration space associated with a single function number. A function may refer to one function in a multi-function device, or to the only function in a single-function device.
A physical function partition adjunct 2860 is employed during initial configuration of the data processing system to facilitate setup of the virtual function partition adjuncts. Note that in the data processing system embodiment depicted in
The virtual function partition adjuncts 2870A, 2871A, 2870B, 2870n implement, in one embodiment, a server virtual adapter device driver, which interfaces with the respective client VIO within the associated logical partition, as well as a device driver for the virtual function on the self-virtualizing input/output device adapter. Each partition adjunct drives the respective virtual function on the adapter similar to the server virtual adapter device driver within a VIOS implementation, such as described above in connection with
Those skilled in the art will note from the above discussion that the programming model described herein allows partition adjunct code developers to write code in a manner that allows typical operating system-kernel-code locking and synchronization primitives to be simplified or avoided. Partition adjuncts may be efficiently written as single-threaded polling state machines, and by controlling the degree of concurrency visible to a particular partition adjunct instance, it is possible to ensure that partition adjunct code may assume a runtime environment that is non-preemptive and single-threaded. For input/output processing work, there is little need for multiple threads to simultaneously execute to service a particular logical partition to virtual function pairing. These properties are feasible because each partition adjunct instance services a single logical partition to virtual function pairing in an isolated/protected environment (that is, a separate dispatchable state protected as described above), and partition adjunct instances can be deployed with overhead low enough to permit a single adjunct instance per logical partition to virtual function pairing, unlike a VIOS implementation.
By providing a partition adjunct 2970A, 2970B . . . 2970n as an interface between each logical partition to queue pair pairing, a virtual input/output interface or driver (client VIO 2915A, 2915B . . . 2915n) can be employed within the respective logical partition. In this case, the logical partition is unaware of the actual type of hardware employed. The partition adjunct may include, for example, a server virtual device driver interfaced to a respective client VIO of the associated logical partition, as well as an appropriate physical adapter device driver for interfacing to the respective queue pair 2951, 2952, 2953 within the self-virtualizing input/output device 2940.
As illustrated in
In contrast,
In this implementation, it is assumed that the partition adjunct instances 3120 are running the same code, since they interface to the same type of virtual function instance within the self-virtualizing input/output device. Thus, memory coalescing of the three partition adjuncts is possible, resulting in only five physical memory pages being employed within physical memory pages 3150, as illustrated in
In this example, the code and read only data reside within physical memory pages 3170 assigned to LPAR1 of the data processing system. The common code and common read only data could alternatively be placed within the memory pages assigned to LPAR2 or the pages assigned to LPAR3. As a further variation, if the partition adjuncts are hypervisor-owned (i.e., initiated), then the code and read only data (as well as the read/write data for the partition adjuncts) would reside in assigned hypervisor pages of the physical memory pages 3150. Each logical partition hardware page table is shown to point to the same pages in physical memory for the partition adjunct code and read only data. Only the page table entries needed for the unique read/write state data for a particular partition adjunct instance are different, as illustrated in
In this example, it is assumed that the partition adjunct instances differ only in the active state they maintain to service their particular logical partition to virtual function pair. Consequently, the hypervisor extracts memory savings by avoiding replication of the common code and read-only data memory areas. The hypervisor may accomplish this in a number of ways, for example, by explicitly recognizing that two partition adjuncts instances are the same, and thus setting up their memory mapping so as to assure maximal sharing, or by using transparent memory coalescing to induce this sharing.
One or more aspects of the present invention can be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has therein, for instance, computer readable program code means or logic (e.g., instructions, code, commands, etc.) to provide and facilitate the capabilities of the present invention. The article of manufacture can be included as a part of a computer system or sold separately.
One example of an article of manufacture or a computer program product incorporating one or more aspects of the present invention is described with reference to
A sequence of program instructions or a logical assembly of one or more interrelated modules defined by one or more computer readable program code means or logic direct the performance of one or more aspects of the present invention.
Although various embodiments are described above, these are only examples.
Moreover, an environment may include an emulator (e.g., software or other emulation mechanisms), in which a particular architecture or subset thereof is emulated. In such an environment, one or more emulation functions of the emulator can implement one or more aspects of the present invention, even though a computer executing the emulator may have a different architecture than the capabilities being emulated. As one example, in emulation mode, the specific instruction or operation being emulated is decoded, and an appropriate emulation function is built to implement the individual instruction or operation.
In an emulation environment, a host computer includes, for instance, a memory to store instructions and data; an instruction fetch unit to fetch instructions from memory and to optionally, provide local buffering for the fetched instruction; an instruction decode unit to receive the instruction fetch unit and to determine the type of instructions that have been fetched; and an instruction execution unit to execute the instructions. Execution may include loading data into a register for memory; storing data back to memory from a register; or performing some type of arithmetic or logical operation, as determined by the decode unit. In one example, each unit is implemented in software. For instance, the operations being performed by the units are implemented as one or more subroutines within emulator software.
Further, a data processing system suitable for storing and/or executing program code is usable that includes at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements include, for instance, local memory employed during actual execution of the program code, bulk storage, and cache memory which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/Output or I/O devices (including, but not limited to, keyboards, displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives and other memory media, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the available types of network adapters.
The capabilities of one or more aspects of the present invention can be implemented in software, firmware, hardware, or some combination thereof. At least one program storage device readable by a machine embodying at least one program of instructions executable by the machine to perform the capabilities of the present invention can be provided.
The flow diagrams depicted herein are just examples. There may be many variations to these diagrams or the steps (or operations) described therein without departing from the spirit of the invention. For instance, the steps may be performed in a differing order, or steps may be added, deleted, or modified. All of these variations are considered a part of the claimed invention.
Although embodiments have been depicted and described in detail herein, it will be apparent to those skilled in the relevant art that various modifications, additions, substitutions and the like can be made without departing from the spirit of the invention and these are therefore considered to be within the scope of the invention as defined in the following claims.
This application claims priority from U.S. Patent Provisional Application No. 60/953,512, filed Aug. 2, 2007, entitled “Partition Adjunct Environment”, the entirety of which is hereby incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5432929 | Escola et al. | Jul 1995 | A |
5555385 | Osisek | Sep 1996 | A |
5729723 | Wada et al. | Mar 1998 | A |
6256704 | Hlava et al. | Jul 2001 | B1 |
6473655 | Gould et al. | Oct 2002 | B1 |
6629162 | Arndt et al. | Sep 2003 | B1 |
6892383 | Arndt | May 2005 | B1 |
6901497 | Tashiro et al. | May 2005 | B2 |
6970990 | Rogers et al. | Nov 2005 | B2 |
7073002 | Armstrong et al. | Jul 2006 | B2 |
7076634 | Lambeth et al. | Jul 2006 | B2 |
7296133 | Swanberg | Nov 2007 | B2 |
7617377 | Moertl et al. | Nov 2009 | B2 |
7802081 | Schultz et al. | Sep 2010 | B2 |
8010763 | Armstrong et al. | Aug 2011 | B2 |
20020016892 | Zalewski et al. | Feb 2002 | A1 |
20030037089 | Cota-Robles et al. | Feb 2003 | A1 |
20040194096 | Armstrong et al. | Sep 2004 | A1 |
20040215917 | Lambeth et al. | Oct 2004 | A1 |
20050027941 | Wang et al. | Feb 2005 | A1 |
20050071841 | Hoflehner et al. | Mar 2005 | A1 |
20050125802 | Wang et al. | Jun 2005 | A1 |
20060020943 | Boutcher et al. | Jan 2006 | A1 |
20060047959 | Morais | Mar 2006 | A1 |
20060064523 | Moriki et al. | Mar 2006 | A1 |
20060090058 | Chng et al. | Apr 2006 | A1 |
20060095690 | Craddock et al. | May 2006 | A1 |
20060143417 | Poisner et al. | Jun 2006 | A1 |
20060195663 | Arndt et al. | Aug 2006 | A1 |
20060209863 | Arndt et al. | Sep 2006 | A1 |
20060230219 | Njoku et al. | Oct 2006 | A1 |
20060248527 | Jaeckel et al. | Nov 2006 | A1 |
20060250945 | Fernandes et al. | Nov 2006 | A1 |
20060259732 | Traut et al. | Nov 2006 | A1 |
20060282624 | Yokota | Dec 2006 | A1 |
20060294288 | Seth et al. | Dec 2006 | A1 |
20070050764 | Traut | Mar 2007 | A1 |
20070169005 | Drepper | Jul 2007 | A1 |
20070169024 | Drepper | Jul 2007 | A1 |
20070180210 | Thibadeau | Aug 2007 | A1 |
20070220246 | Powell et al. | Sep 2007 | A1 |
20070271565 | Tirumalai et al. | Nov 2007 | A1 |
20080016337 | Morgan et al. | Jan 2008 | A1 |
20080091855 | Moertl et al. | Apr 2008 | A1 |
20080126690 | Rajan et al. | May 2008 | A1 |
20080307425 | Thipathi | Dec 2008 | A1 |
20090037682 | Armstrong et al. | Feb 2009 | A1 |
20090037906 | Armstrong et al. | Feb 2009 | A1 |
20090037907 | Armstrong et al. | Feb 2009 | A1 |
20090037908 | Armstrong et al. | Feb 2009 | A1 |
Number | Date | Country |
---|---|---|
1548589 | Jun 2005 | EP |
1691287 | Aug 2006 | EP |
2008041093 | Feb 2008 | JP |
Entry |
---|
“Single Root I/O Virtualization and Sharing Specification—Revision 1.0”, PCI-SIG (Sep. 11, 2007). |
“Xen: Enterprise Grade Open Source Virtualization, Inside Xen 3.0—A XenSource White Paper”, XenSource, Inc., (pp. 1-4) (Jun. 1, 2006). |
Office Action for U.S. Appl. No. 12/111,041 (U.S. Patent Publication No. 2009/0037682), dated Nov. 19, 2010. |
Office Action for U.S. Appl. No. 12/110,953 (U.S. Patent Publication No. 2009/0037907 A1), dated Aug. 11, 2011. |
Office Action for U.S. Appl. No. 12/110,982 (U.S. Patent Publication No. 2009/0037908 A1), dated Sep. 29, 2011. |
Office Action for U.S. Appl. No. 12/110,923 (U.S. Patent Publication No. 2009/0037906 A1), dated Oct. 6, 2011. |
Wang et al., “Helper Threads via Virtual Multithreading on an Experimental Itanium 2 Processor-based Platform”, ASPLOS'04, Oct. 9-13, 2004 (pp. 144-155). |
Armstrong et al., Notice of Allowance for U.S. Appl. No. 12/110,953, filed Apr. 28, 2008 (U.S. Patent Publication No. 2009/0037907 A1), dated Dec. 29, 2011. |
Armstrong et al., Notice of Allowance for U.S. Appl. No. 12/110,923, filed Apr. 28, 2008 (U.S. Patent Publication No. 2009/0037906 A1), dated Mar. 2, 2012. |
Armstrong et al., Notice of Allowance for U.S. Appl. No. 12/110,982, filed Apr. 28, 2008 (U.S. Patent Publication No. 2009/0037908 A1), dated Mar. 2, 2012. |
Armstrong et al., Notice of Allowance for U.S. Appl. No. 13/441,139, filed Apr. 6, 2013 (U.S. Patent Publication No. 2012/0210044 A1), dated Mar. 25, 2013. |
Office Action for U.S. Appl. No. 12/110,953 (U.S. Patent Publication No. 2009/0037907 A1), dated Nov. 17, 2011. |
Translations of portions of Nov. 15, 2011 Office Action that apply, discuss, or otherwise mention the attached references. |
Number | Date | Country | |
---|---|---|---|
20090037941 A1 | Feb 2009 | US |
Number | Date | Country | |
---|---|---|---|
60953512 | Aug 2007 | US |