Patent Application
20160077981
1. Field
The disclosed embodiments relate generally to computer systems, and in particular to memory management units for input/output (IO) devices.
2. Background Art
Computer systems of various types are ubiquitous in modern society, including personal computers (PCs), workstations, servers, various personal digital assistant (PDA) devices, etc. Most, if not all, of these computer systems have implemented memory management functionality for processor accesses to memory. Generally, the memory management functionality has included translating addresses from a virtual address space used by each process to a physical address space that spans the actual system memory, along with various memory protections (e.g., read only, read/write, privilege level requirements, etc.). The memory management functionality has a variety of uses, such as protecting the memory used by each process from unauthorized access by other processes, permitting large virtual spaces to be used by processes even if the physical memory system is not that large, relocation of virtual addresses to available physical memory without the participation of the process, and the like.
Like processor addresses that are frequently translated, addresses used by input/output (IO) devices in computer systems can also be translated. That is, the IO devices can use virtual addresses rather than physical addresses to access memory. Use of virtual addresses rather than physical addresses by devices is preferred in current day systems since it improves the overall security of the system. Use of physical addresses by a rogue device (or a device programmed by a malicious software agent) would result in impeded memory access.
In single operating system (OS) computer systems, such as most PCs, the OS controls access to the IO devices by other processes (applications and OS services). Accordingly, the OS can control which process has access to a given device at any given point in time, and can at least somewhat control the addresses accessed by the device. Virtual machine systems are more complex, as they may have multiple guest OSs running on a virtual machine monitor. In a virtualized system, many applications and IO devices access physical memory through the use of virtual addresses. An IO memory unit is coupled to the IO devices and the system memory, where the IOMMU is configured to translate the virtual address in the device memory request to physical addresses to access the physical system memory.
Modern day computing environments include virtual machine (VM) environments, in which multiple VMs can execute on a single processor system as separate logical operating entities. Typically, these logically separated VMs share common resources of the processor system, such as hardware devices and device drivers. To manage the co-existence of these multiple VMs and to enable exchanging information with common resources and between these VMs, VM environments often use a virtual machine monitor (VMM) or hypervisor.
Security and stability are important issues in most computer systems, and in particular to VM environments. In traditional computer systems, peripheral devices and their associated device drivers have free and unfettered access to memory. Such unfettered access means that a corrupted or malfunctioning device or device driver can write in any location in memory, whether or not that memory location has been set aside for use by that peripheral. Should a memory location set aside for operating system use be overwritten by the malfunctioning device, a system crash will almost inevitably result. Computer system users demand stability, and system crashes due to memory corruption are sought to be minimized.
As can be expected, the attention directed to stability and security aspects requires the use of CPU cycles to provide the additional functionality. Such CPU cycle usage constitutes overhead in the sense that it does not directly contribute to the functionality of the application. Consequently, the additional overhead results in a reduction in efficiency to that achievable without the additional security and stability.
What is needed is a virtual IOMMU that can prevent any improper access in a virtualization environment while offering an improved efficiency over that provided by a normal IOMMU.
In some embodiments, a user-level IO function is described in a virtualized environment. The user-level IO function receives an IO operation from a device driver associated with an IO device. The IO operation has one or more guest virtual addresses. An IO memory management unit (IOMMU) validates the one or more guest virtual addresses as being associated with the IO card. If the one or more guest virtual addresses are associated with the IO card, the IO operation is allowed to propagate to the IO card.
In some embodiments, a method is described for propagating an IO operation from a device driver across to an IO card in a virtualized environment. The IO operation is received from a device driver by a user-level IO function, where the IO operation has one or more guest virtual addresses. An IO memory management unit (IOMMU) validates the one or more guest virtual addresses as being associated with the IO card. If the one or more guest virtual addresses are validated as being associated with the IO card, the IO operation is allowed to propagate to the IO card.
Further embodiments, features, and advantages of the disclosed embodiments, as well as the structure and operation of the various embodiments are described in detail below with reference to the accompanying drawings.
The accompanying drawings, which are incorporated herein and form part of the specification, illustrate the disclosed embodiments and, together with the description, further serve to explain the principles of the disclosed embodiments and to enable a person skilled in the relevant art(s) to make and use the disclosed embodiments.
The features and advantages of the disclosed embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.
By way of background, terms such as virtualization and virtual are used in accordance with their meaning in the computing arts. In particular, virtualization refers to techniques used to hide the physical characteristics of an underlying resource so as to simplify the approach by which systems, applications and end-users can interact with those resources. For example, virtualization allows a single physical resource (e.g., memory) to appear as multiple logical resources. Thus, virtualization techniques permit a single computer to be able to run a number of virtual machines, with each virtual machine appearing to have its full complement of resources available to it, and without any apparent overlap of resources to other virtual machines.
Referring to
As described further below, the IOMMU 26 may include various features to simplify virtualization in the system 10. The description below will refer to a virtual machine monitor (VMM) that manages the virtual machines (scheduling their execution on the underlying hardware), controls access to various system resources, etc. It is noted that VMMs are also sometimes referred to as hypervisors. In the illustrated embodiment, processor(s) 12 is executing software in a virtualized environment. Accordingly, three virtual machines 100A, 100B, and 100C (e.g., VM 1-3) and a VMM 106 are shown. The number of virtual machines 100 in a given embodiment may vary, and may dynamically change during use as virtual machines are started and stopped by a user. In the illustrated embodiment, the virtual machine 100A includes one or more guest applications 102 and a guest operating system (OS) 104. The OS 104 is referred to as a “guest” OS, since the OS 104 controls the virtual machine 100 created for it by the VMM 106, rather than the physical hardware of the system 10. Similarly, the VM 100B and VM 100C may also each include one or more guest applications 102 and a guest OS 104.
Generally, the applications 102 in the virtual machines 100 use a guest virtual address space, and thus guest virtual addresses (GVA). The guest OS 104 in each virtual machine 100A, 100B, 100C may manage mappings of the GVA to guest “physical” addresses (GPA) in the virtual machine 100A, 100B, 100C. If the guest OS 104 were running directly on the system 10 hardware, with no VMM, the physical addresses generated by the guest OS 104 would indeed be the system physical addresses (SPA) of the memory locations in the system 10. However, in the virtual machine environment, the VMM 106 may manage the mappings from GPA to SPA. Thus, when processor 12 performs memory requests, the guest OS 104 may manage mappings of GVA to GPA (which may be further mapped to SPA by the VMM 106).
As illustrated in
Generally, the IO devices 22 may be configured to issue memory requests, such as memory read and write requests, to access memory locations in the memory 20 and in some cases, translation requests. The memory requests may be part of a direct memory access (DMA) read or write operation, for example. The DMA operations may be initiated by software executed by the processors 12, programming the IO devices 22 directly or indirectly to perform the DMA operations. Depending on the address space in which the software executing on the processor is running, the IO devices 22 may be provided with addresses corresponding to that address space to access the memory 20. For example, a guest application (e.g., App 102) executing on processor 12 may provide an IO device 22 with GVAs, while a guest OS executing on processor 12 (e.g., OS 104) may provide GPAs to the IO devices 22. In either case, when the IO device 22 requests a memory access, the guest addresses may be translated by the IOMMU 26 to corresponding SPAs to access the memory, and the system physical addresses may be provided to the memory controller 18 for access. That is, the IOMMU 26 may modify the memory requests sourced by the IO devices 22 to change (i.e., translate) the received address in the request to an SPA, and the memory request may be forwarded to the memory controller 18 to access the memory 20.
In various embodiments, the IOMMU 26 may provide one-level, two-level, or no translations depending on the type of address it receives from the IO device. More particularly, the IOMMU 26 may perform one-level nested translations or two-level guest translations. That is to say, the IOMMU 26 may provide both GPA to SPA translations (one-level), and GVA to SPA translations (two-level). Thus, as mentioned above, a guest application 102 may provide GVA addresses directly to an IO device 22 when requesting memory accesses, thereby making conventional VMM interception and translation unnecessary. This functionality may allow advanced computation architectures, such as compute offload, user-level IO, and accelerated IO devices, to be used more seamlessly in virtualized systems. It is noted that although one-level, two-level, or no translations are described, it is contemplated that in other embodiments, additional levels of address space may be used. In such embodiments, additional levels of translation (i.e., multilevel translations) may be performed by IOMMU 26 to accommodate the additional address spaces.
As described in greater detail below, the IOMMU 26 has a way of recognizing what type of address it is receiving in a given request. Accordingly, in embodiments in which IO devices 22 are coupled to IOMMU 26 via standard peripheral buses, such as PCI express (PCIe) interconnects, for example, a process address space identifier (PASID), may be sent to the IOMMU 26 using a transaction layer protocol (TLP) prefix.
The IOMMU 26 may use a number of data structures, such as one or more sets of IO translation tables 36 stored in the memory 20, to translate the addresses of memory and translation requests from the IO devices 22. Generally, IO translation tables 36 may be tables of translation data that can be used to translate addresses from one type to another. The IO translation tables 36 may store the translation data in any fashion. For example, in one embodiment, the IO translation tables 36 may include page tables similar to those defined in the x86 and AMD64™ instruction set architectures. Depending on the translation level, various subsets of the guest virtual address bits or guest physical address may be used to index levels of the IO translation tables 36, and each level may either be the end of translation (i.e., storing a real page number for the translation) or may point to another table (indexed by another set of address bits). The page may be the unit of translation (i.e., each address in the virtual page translates to the same physical page). Pages may have varying sizes, e.g., from 4 kilobytes up to Megabytes or even Gigabytes.
Additionally, the IO translation tables 36 may include a device table (e.g., as shown in
Specifically, the IOMMU 26 illustrated in
To facilitate more rapid translations, the IOMMU 26 may cache some translation data. For example, the cache 30 may be a form of cache similar to a TLB (or IOTLB), which caches the result of previous translations, mapping guest virtual and guest physical page numbers to system physical page numbers and corresponding translation data. If a translation is not found in the cache 30 for the given memory request, the table walker 28 may be invoked. In various embodiments, the table walker 28 may be implemented in hardware, or in a microcontroller or other processor and corresponding executable code (e.g., in a read-only memory (ROM) in the IOMMU 26). Additionally, other caches may be included to cache page tables, or portions thereof, and/or device tables, or portions thereof, as part of cache 30. Accordingly, the IOMMU 26 may include one or more memories to store translation data that is read from, or derived from, translation data stored in the memory 20.
As noted above, when application 210 initiates an input/output (IO) operation, application 210 would call kernel 230, which would in turn call hypervisor 250. The interaction of application 210 with kernel 230 occurs via an IO system call (syscall) 270 from device driver (or driver) 220. Here, the operating system copies the relevant IO data from user space to kernel space, and dispatches the data to driver 220. Application 210 exists in the user domain, and the handoff from the user domain to the kernel domain involves the use of a context switch (not illustrated). The context switch permits kernel 230 to store the state (or context) of application 210 at the time of the IO syscall 270. Thus, the stored state permits a restore of the state at a later point in time. Consequently, kernel 230 and application 210 can continue other processes while waiting for the requested IO process to complete, and provide the requested action or information. However, the context switch is considered expensive, as it requires on the order of 5000 to 20000 instructions to execute. These instructions are considered overhead in that none of these instructions provide direct value to the calling application 210. For example, in the case of a spreadsheet application, none of these instructions directly contribute to an updated cell value in the spreadsheet file.
Continuing to refer to
To further accentuate the point, note that the IO syscall 270 is created by application 210 (running in the guest virtual address (GVA) space) and is sent to the kernel 230 (running in the guest physical address (GPA) space). IO syscall 270 uses processing to switch from user to kernel mode safely, and then kernel 230 validates the details of the IO request, such as the buffer addresses. Next, kernel 230 converts IO syscall 270 into a form for use by hypervisor 265, namely IO hypercall 275. Kernel 230 passes IO hypercall 275 to hypervisor 265 indirectly (e.g., via traps), but in any event, both approaches require one or more “world switches” with the associated overhead. After vetting the calling arguments and converting the addresses from GPA to system physical address (SPA) space, hypervisor 265 passes the once again reformed IO operation 280 to IO card 260. As noted, significant overheads may be incurred as IO syscall 270 and IO hypercall 275 are passed to the next level down.
In further reference to
SR-IOV functionality provides a standardized approach to the sharing of IO physical devices in a virtualized environment. In particular, SR-IOV provides a mechanism by which a single root (SR) function, such as a single IO device, can appear to be multiple separate physical devices for the multiple virtual machines. For example, the IO device can be configured by hypervisor 350 to appear in the PCI configuration space as multiple functions, with each function having its own configuration space. Thus, the independent configuration space for each virtual machine enables data movement to bypass involvement of hypervisor 350.
As noted above, when application 310 initiates an input/output (IO) operation, application 310 calls kernel 330. The interaction of application 310 with kernel 330 occurs via an IO system call (syscall) 375 from device driver (or driver) 320. Here, the operating system copies the relevant IO data from user space to kernel space, and dispatches the data to driver 320. Application 310 exists in the user domain, and the handoff from the user domain to the kernel domain continues to involve a context switch (not illustrated). As noted above, the context switch permits kernel 330 to store the state (context) of application 310 at the time of the IO syscall 375. Thus, the stored state permits a restore of the state at a later point in time. Consequently, kernel 330 and application 310 can continue other processes while waiting for the requested IO process to complete, and provide the requested action or information.
Continuing to refer to
In one example, IOMMU 370 has been added to the system as part of the hardware support. Hypervisor 350 can configure IOMMU 370 to perform validation operations previously done by hypervisor software modules, with the result that IO hypercall 275 is completely eliminated. In one example, IO syscall 375 is identical in form and operation to IO syscall 270 that was illustrated in
As before, at a later point in time, IO card 360 receives the information provided by the IO device in response to the IO operation. Such a return is provided through the use of a completion interrupt 385, which alerts hypervisor 350 to the availability of the IO information. The data received in completion interrupt 385 is assigned to the appropriate guest OS (in this case kernel 330), and is forwarded by SR-IOV virtual function 340 to driver 320 via a virtual interrupt to the driver. In turn, kernel 330 restores the state associated with the original IO operation. Driver 320 copies the completion report as required, and signals the completion of the IO operation to calling application 310 by initiating a return call 395. Calling application 310 then processes the completion report.
As noted above, SR-IOV functionality provides a standardized approach to the sharing of IO physical devices in a virtualized environment. In particular, SR-IOV provides a mechanism by which a single root (SR) function such as a single IO device can appear to be multiple separate physical devices for the multiple virtual machines. For example, the IO device can be configured by hypervisor 450 to appear in the PCI configuration space as multiple functions as multiple functions, with each function having its own configuration space. Thus, the independent configuration space for each virtual machine enables data movement to bypass involvement of hypervisor 450. The embodiment illustrated in
With continuing reference to
In this virtualized environment, the IOMMU 470 supports two “stages” of translation of addresses, and thus the IO card 460 (and its associated external IO device) can be assigned directly to application 410 within a guest OS (kernel) 430.
Thus, in this architecture, there is neither a handoff from the user domain to the kernel domain that involves a context switch, nor a handoff from the kernel domain to the hypervisor domain that involves the world switch. Accordingly, this architecture eliminates the overhead of both the context switch and the world switch and their respective inefficient overhead contributions. SR-IOV virtual function 440 is “fenced” in by IOMMU 470. As above, the term “fenced” in refers to the range of valid addresses to which SR-IOV virtual function 440 can validly address in IO operation 475. IO operation 475 contains one or more guest virtual addresses as being associated with the IO card. IOMMU 470 checks these guest virtual addresses as being within a valid range. If the guest virtual addresses are within a valid range, IO operation 475 can propagate from driver 420 to IO card 460. SR-IOV virtual function 440 can be implemented in either software or hardware, or a hybrid of hardware and software. However, an implementation in hardware accelerates the response time of SR-IOV virtual function 440, and in addition provides increased reliability as well as a relatively bug-free implementation.
In one example, because hypervisor 450 and kernel 430 can program IOMMU 470 hardware modules, application 410 can issue IO operation 475 directly to IO card 460. The previous user-kernel and kernel-hypervisor overheads are substantially eliminated because IOMMU 470 hardware modules do what the software previously did. This means that application 410 will actually run faster than a system with native hardware or a virtualized system (i.e., essentially faster than 100% performance). For example, consider a system in which application 410 is a web server and IO card 460 is a network interface card (NIC). The system illustrated in
As before, at a later point in time, IO card 460 receives the information provided by the IO device in response to the IO operation. Such a return is provided through an interrupt which is routed to the guest OS or kernel 430. As a result, kernel 430 posts a virtual interrupt to driver 420. The data received in the completion interrupt 485 is assigned to the appropriate guest OS (in this case kernel 430), and is forwarded by SR-IOV virtual function 440 to driver 420 via a virtual interrupt to the driver. Driver 420 copies the completion report as required, and signals the completion of the IO operation to calling application 410 by initiating a return call 480. Calling application 410 then processes the completion report.
In embodiments of the present invention, the underlying functionality is pushed down (i.e., performed) into hardware. By contrast, other approaches use the more inefficient and unreliable software approach. For example, paravirtualization attempts the same objectives as embodiments of the present invention, but paravirtualization uses software instead of hardware. In paravirtualization, programmers can modify applications and system software to provide improved efficiency, but this solution does not provide the same level of trustworthy and reliable operation that a hardware-based solution provides. An early attempt at such a software based solution was “WinSock Direct.” As known by those of ordinary skill in the art, software is vulnerable to both bugs as well as attacks by viruses. Although bugs exist in hardware, they are far less common than bugs in software, and therefore far less of a problem. Unlike software, which is readily amenable to change, hardware functionality is defined by gates and wires, and therefore cannot be modified by software attempting to subvert proper operation.
The process begins at step 510. In step 510, an IO operation is received by a user-level IO function in a virtualized environment. The IO operation originates in a device driver associated with an IO device, and has one or more guest virtual addresses. In an embodiment, IO operation 475 originates in driver 420.
In step 520, the one or more guest virtual addresses are validated as being associated with the IO card. Validation is performed by an IO memory management unit (IOMMU). In an embodiment, validation of the guest virtual addresses in IO operation 475 is performed by IOMMU 470.
In step 530, the IO operation is propagated to an IO card associated with the IO device if the one or more guest virtual addresses are validated as being associated with the IO card. In an embodiment, IO operation 475 is propagated to IO card 460.
In step 540, method 500 ends.
The embodiments described, and references in the specification to “some embodiments,” indicate that the embodiments described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with particular embodiments, it is understood that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
Embodiments of the invention may be implemented in hardware, firmware, software, or any combination thereof. Embodiments of the invention may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. For example, a virtual IOMMU may be emulated by instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others. Further, firmware, software, routines, instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.
The embodiments have been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.
The foregoing description of the specific embodiments will so fully reveal the general nature of the inventive subject matter such that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the inventive subject matter. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.
