Patent Grant
12321769
Computing devices can utilize communication networks to exchange data. Companies and organizations operate computer networks that interconnect a number of computing devices to support operations or provide services to third parties. The computing systems can be located in a single geographic location or located in multiple, distinct geographic locations (e.g., interconnected via private or public communication networks). Specifically, data centers or data processing centers, herein generally referred to as “data centers,” may include a number of interconnected computing systems to provide computing resources to users of the data center. The data centers may be private data centers operated on behalf of an organization, or public data centers operated on behalf, or for the benefit of, the general public.
To facilitate increased utilization of data center resources, virtualization technologies allow a single physical computing device to host one or more instances of virtual machines that appear and operate as independent computing devices to users of a data center. With virtualization, the single physical computing device can create, maintain, delete, or otherwise manage virtual machines in a dynamic manner. In turn, users can request computing resources from a data center, such as single computing devices or a configuration of networked computing devices, and be provided with varying numbers of virtual machine resources.
Virtual machines can be provisioned with multiple components of resources. For example, virtual machines can be provisioned with memory, a disk, a graphical processing unit, etc. Based on the resources provisioned for the virtual machine, a user may configure the virtual machine to implement desired functionality (e.g., to conduct computations). Specifically, users can provide a plurality of code for execution using the provisioned resources. Execution of the plurality of code may cause changes to the state of the provisioned resources. For example, execution of the code may modify provisioned memory for the virtual machine. Due to the virtualized and/or dynamic nature of the virtual machines, the changes to the state may not be maintained for use in executing additional code.
Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure.
Generally described, aspects of the present disclosure relate to the execution of a code module based on a “snapshot” of state information of components for execution of the code module in an on-demand code execution system. The on-demand code execution system enables rapid execution of code modules, which may be supplied by users of the on-demand code execution system. More specifically, embodiments of the present disclosure relate to improving the performance of an on-demand code execution system that is implemented using various computing resources. As described in detail herein, the on-demand code execution system may provide a network-accessible service enabling users to submit or designate computer-executable code (threads, programs, etc., composed in any of a variety of programming languages) to be executed by virtual machine instances on the on-demand code execution system. Users may submit the code module as a notebook cell or a code cell. For example, a user may define a code cell of a notebook where the code cell (or notebook cell) includes one or more statements in a programming language. Each code module may define a set of code for execution. The set of code may include any code that requires additional data (e.g., state information) for execution of the code. Each set of code may also include code associated with a plurality of code classes, a plurality of code functions, a plurality of code languages, or a plurality of code identifiers. Further, each code module on the on-demand code execution system may define a “task” to execute the set of code and implement specific functionality corresponding to that task when executed on a virtual machine instance of the on demand code execution system. For example, a user may submit a script in a specific programming language (e.g., the PYTHON™ language) that, when executed, implements network-based processing for a user-facing application (e.g., a mobile device “app”). Individual implementations of the task on the on-demand code execution system may be referred to as an “execution” of the task (or a “task execution”). The on-demand code execution system can further enable users to trigger execution of a task based on a variety of potential events, such as detecting new data at a network-based storage system, transmission of an application programming interface (“API”) call to the on-demand code execution system, or transmission of a specially formatted hypertext transport protocol (“HTTP”) packet to the on-demand code execution system. Thus, users may utilize the on-demand code execution system to execute any specified executable code “on-demand,” without requiring configuration or maintenance of the underlying hardware or infrastructure on which the code is executed. Further, the on-demand code execution system may be configured to execute tasks in a rapid manner (e.g., in under 100 milliseconds [ms]), thus enabling execution of tasks in “real-time” (e.g., with little or no perceptible delay to an end user).
The on-demand code execution system may thus allow users to execute code in a serverless environment (e.g., one in which the underlying server is not under user control). The serverless environment then enable the user to submit “calls” to execute that script, at which point the system will securely execute the script to provide the desired functionality. Unlike some other network-based services, a serverless environment can remove the need for a user to maintain or configure a computing device, either virtual or physical, to support code execution. It is this lack of need for a user to maintain a device that leads to the “serverless” moniker, though of course the serverless environment itself, as opposed to individual users, likely maintains servers to support code execution. Serverless environments can be particularly well suited for processes with varying demand, as the serverless environments can rapidly scale up and down the computing resources used to service such processes. In comparison to traditional systems using dedicated servers (physical or virtual), serverless environments often provide much higher efficiency with respect to computing resources used. The serverless environment may enable the stateless execution of code modules. The on-demand code execution system may dynamically spin up virtual machine instances based on receiving a request to implement a code module. The on-demand code execution system may utilize the virtual machine instance to execute the code module. During execution of the code module, the virtual machine instance may generate state information based on executing the code module. For example, the state information may identify (e.g., indicate) the modification of a modified state of components (e.g., memory, a disk, a graphical processing unit) of the virtual machine instance where the state of the components is modified due to the execution of the code module. However, after executing the code module, the on-demand code execution system may tear down (e.g., release) the virtual machine instance and release the state information. Therefore, the state information from the execution of the code module may not be retained for the execution of subsequent code modules. Instead, the on-demand code execution system may statelessly execute each code module.
As noted above, one issue that may be of particular concern in serverless environments is that of stateful execution of code as users may wish to retain state information generated from the execution of a code module. For example, the execution of a code module may cause the state of components of the virtual machine instances to be modified (e.g., the state information is changed). As the code module is executed in an on-demand code execution system, the modified state may not be retained. A user may wish to retain this modified state in order to perform further operations. Further, the state information generated as a result of the execution of the code module may contain important information that the user may desire to retain for further operations or the user may desire to execute additional code modules based on the previously generated or modified state information (e.g., the user may desire to undo the changes to the state information, redo the changes, fork the state information, merge the state information, select information from the state information, etc.). Therefore, the stateless execution of the code modules can lead to inadequate user experiences as the user may be unable to iteratively execute code modules against saved state information and can lead inaccurate results. Instead, each code module may be executed in view of particular base state information (e.g., no state information) and any state information generated by the execution of a code module may be released upon release of the virtual machine instance.
Further, the execution of a code module using serverless computing may be unable to guarantee routing of the code module to any particular serverless environment. For example, two calls to execute a code module may go to different serverless environments. Therefore, if the state information is retained for a particular code module executed in a first serverless environment, the execution of another code module may occur in a second serverless environment that does not have access to the state information. Instead, each code module may be executed in different serverless environments that may not have access to the previously generated state information. Further, while traditional snapshotting may enable the state of an executing machine to be paused and resumed, the state information is paused and may be tied to a particular module. Therefore, the state information may be coupled to a particular code module and may not be forked, versioned, etc.
In some embodiments, a user may utilize web applications that allow users to create and share live code for deployment (e.g., code modules). The code may be implemented and deployed via a “notebook.” Notebooks let users provide new code modules for execution against previously generated state information. Such notebooks include a plurality of code modules or code cells. The code cells may each incrementally change state information of an execution environment (e.g., memory, disks, graphical processing units (“GPUs”), etc.). The code cells of the notebooks may be implemented using hosted hardware. For example, one or more compute instances and one or more containers. However, the use of hosted hardware may be inefficient. For example, the request rate to the hosted hardware may be slow and inefficient. Further, the cost associated with paying for and maintaining the hosted hardware may be extensive. Further still, while the state information generated by code cells may be retained, a user may be unable to undo or redo the modifications to the state information caused by the execution of the code cells. Instead, while the execution of the code cells may be undone and redone, the modifications to the state information may not be redone or undone. Further, while the execution system may retain the state information, the execution system may not retain different versions or iterations of the state information. This can lead to an inadequate performance and is generally detrimental to the goal of enabling users to dynamically manage the state information generated by the execution of code cells.
Embodiments of the present disclosure address these problems by enabling an on-demand code execution system to incrementally “snapshot” state information of components of the virtual machine, the state information capturing modifications to the state of the components caused by the execution of code modules. Such an incremental snapshot of the state information retains the advantages offered by the use of notebooks by enabling users to provide code modules for execution against previously generated state information while retaining the advantages of an on-demand code execution system. Further, the incremental snapshotting can reduce or eliminate performance issues, due, for example, to a lack of saved state information for use in execution of multiple code modules. More specifically, embodiments of the present disclosure enable the incremental snapshot of state information generated by the execution of a code module by a virtual machine instance, prior to the on-demand code execution system tearing down the virtual machine instance. Further, the incremental snapshot of the state information may be retained after the virtual machine instance is torn down. This can enable the state information to be utilized for the execution of additional code modules. Further, the code modules may be implemented in a serverless environment while achieving increased functionality, thereby maintaining the benefit offered by the execution of the code module using virtual resources and maintaining the benefit of saved state information. By capturing the incremental snapshots of state information, the particular code module associated with the state information may be decoupled from the state information. Therefore, multiple code modules may be used to transform or manipulate the state information in different ways (e.g., the state information can be versioned or forked, changes to the state information can be undone or redone, etc.) Thus, the on-demand code execution system may incrementally snapshot state information generated or modified by the execution of code modules.
Further, by snapshotting the state information, statefulness can be retained in serverless execution of code modules. The snapshots of the state information capture the state information without requiring execution of the code modules in a stable environment. While traditional cloud computing environments may utilize stable environments to execute code modules, the incremental snapshot of the state information enables state information to be retained in a serverless execution of code modules. Further, the state information may be decoupled from a specific code module. Thus, the state information can be utilized across multiple code modules executed in separate serverless environments. In order to retain the statefulness, each request to execute a code module can be accompanied with an identifier of a snapshot (e.g., a pointer to a snapshot). The snapshot may represent state information that the code module should use for execution. The request for execution of the code module can be satisfied by provisioning an execution environment with a snapshot holding the state information, and executing the code in the provisioned environment. Therefore, requests for execution may be directly tied to specific state information, relieving a need to separately configure an environment with state information prior to executing a code module in that environment. Thus, the end user experience may be simplified (as only a single request may be required) and a serverless compute system may maintain the ability to distribute requests according to internal load balancing, rather than to any particular environment. Moreover, by decoupling code modules from specific state information, an end user may be enabled to “mix and match” code and state according to their requirements. For example, each request to execute a code module may specify particular state information to be provisioned into an execution environment for the code module, which particular state information may have been generated by execution of a prior code module. In this manner, state information can be passed between code modules, without requiring the state information itself to be passed in a call to invoke the code module, and without strongly coupling all executions of a code module to particular state information.
Each virtual machine instance of the on-demand code execution system may be associated with state information. For example, the state information may identify a state of one or more components (e.g., memory, disks, GPUs, etc.) of the virtual machine instance. The on-demand code execution system may execute a plurality of code modules. The execution of each code module may modify the state of the one or more components of the virtual machine instance executing the particular code module. For example, the execution of a particular code module may modify memory of a particular virtual machine instance used for the execution of the code module. Based on the execution of the code module, the on-demand code execution system can “snapshot” the state of the one or more components to generate a snapshot of the state information. The on-demand code execution system may then release the virtual machine instance and the one or more components of the virtual machine instance. Further, the on-demand code execution system may incrementally snapshot the state of the one or more components to generate incremental or versioned snapshots of the state information. For example, the on-demand code execution system may generate incremental snapshots that identify differences between first state information and second state information. Therefore, the incremental snapshots may store a minimum amount of data to identify differences, or a delta, between first state information and second state information and may not identify the full second state information. The incremental or versioned snapshots may be stored as chunks of state information. Thus, the state information generated by the execution of code modules may be stored as chunks for execution of additional code modules.
As described below, to utilize the snapshot of the state information, a code module may be executed based on the snapshot of the state information. The chunks of the state information may be “lazily” loaded into an execution environment, by quickly providing a minimum portion of the state information needed to begin execution of code, and providing additional portions of the state information on-demand for the code execution. For example, the minimum portion of the state information may identify the differences between base state information (e.g., original state information) and updated state information (e.g., state information modified by execution of a code module). More specifically, a request to execute a code module may be satisfied by provisioning an execution environment with access to a snapshot of state information, without actually transferring the entire state information to a local storage drive for the environment. Instead, the state information may be made available via a file system that operates to selectively retrieve portions of the state information as they are read by the code execution. For example, a host computing device that is hosting an execution environment may be configured to provide a Filesystem in User Space (FUSE) storage device that—from the view of the execution environment—contains the state information. On reads to the FUSE storage device, a local FUSE agent may selectively retrieve any required portions of the state information and make the read portion of the state information available on the FUSE storage device. Thus, from the point of view of the execution environment, complete local access to the state information is provided. However, because the state information is “lazily” loaded, code execution can begin before the entire state information is transferred to the execution environment. Indeed, if the environment never requires access to a portion of the state information, that portion need never be transferred to the environment. As such, the latency to execute code is reduced. Various embodiments for loading the state information are described in more detail in U.S. patent application Ser. No. 17/105,250, entitled “LOW LATENCY ACCESS TO DATA SETS USING SHARED DATA SET PORTIONS,” and filed Nov. 25, 2020 (the “250 Patent Application”), the entirety of which is hereby incorporated by reference.
While a virtual machine executing an operating system is described herein as one example of an execution environment, other execution environments are also possible. For example, tasks or other processes may be executed within a software “container,” which provides a runtime environment without itself providing virtualization of hardware. Containers may be implemented within virtual machines to provide additional security, or may be run outside of a virtual machine instance.
As will be appreciated by one of skill in the art in light of the present disclosure, the embodiments disclosed herein improve the ability of computing systems, such as on-demand code execution systems, to retain state information generated or modified based on the execution of a code module. Moreover, the presently disclosed embodiments address technical problems inherent within computing systems; specifically, the difficulties of retaining state information for an on-demand code execution environment. These technical problems are addressed by the various technical solutions described herein, including incrementally snapshotting the state information generated by the execution of code modules and executing additional code modules based on the incremental snapshots. Thus, the present disclosure represents an improvement on existing data processing systems and computing systems in general.
The foregoing aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following description, when taken in conjunction with the accompanying drawings.
The illustrative environment 100 further includes one or more auxiliary services 106, which can interact with the serverless code execution environment 110 to implement desired functionality on behalf of a user. Auxiliary services 106 can correspond to network-connected computing devices, such as servers, which generate data accessible to the serverless code execution system 110 or otherwise communicate to the serverless code execution system 110. For example, the auxiliary services 106 can include web services (e.g., associated with the user computing devices 102, with the serverless code execution system 110, or with third parties), databases, really simple syndication (“RSS”) readers, social networking sites, or any other source of network-accessible service or data source. In some instances, auxiliary services 106 may be invoked by code execution on the serverless code execution system 110, such as by API calls to the auxiliary services 106. In some instances, auxiliary services 106 may be associated with the serverless code execution system 110, e.g., to provide billing or logging services to the serverless code execution system 110. In some instances, auxiliary services 106 actively transmit information, such as API calls or other task-triggering information, to the serverless code execution system 110. In other instances, auxiliary services 106 may be passive, such that data is made available for access by the serverless code execution system 110. For example, components of the serverless code execution system 110 may periodically poll such passive data sources, and trigger execution of code within the serverless code execution system 110 based on the data provided. While depicted in
The client devices 102, auxiliary services 106, and serverless code execution system 110 may communicate via a network 104, which may include any wired network, wireless network, or combination thereof. For example, the network 104 may be a personal area network, local area network, wide area network, over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or combination thereof. As a further example, the network 104 may be a publicly accessible network of linked networks, possibly operated by various distinct parties, such as the Internet. In some embodiments, the network 104 may be a private or semi-private network, such as a corporate or university intranet. The network 104 may include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. The network 104 can use protocols and components for communicating via the Internet or any of the other aforementioned types of networks. For example, the protocols used by the network 104 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queue Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), and the like. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art and, thus, are not described in more detail herein.
The serverless code execution system 110 is depicted in
Further, the serverless code execution system 110 may be implemented directly in hardware or software executed by hardware devices and may, for instance, include one or more physical or virtual servers implemented on physical computer hardware configured to execute computer executable instructions for performing various features that will be described herein. The one or more servers may be geographically dispersed or geographically co-located, for instance, in one or more data centers. In some instances, the one or more servers may operate as part of a system of rapidly provisioned and released computing resources, often referred to as a “cloud computing environment.”
In the example of
In
To enable interaction with the serverless code execution system 110, the system 110 includes multiple frontends 120, which enable interaction with the serverless code execution system 110. In an illustrative embodiment, the frontends 120 serve as a “front door” to the other services provided by the serverless code execution system 110, enabling users (via user computing devices 102) to provide, request execution of, and view results of computer executable source code. The frontends 120 include a variety of components to enable interaction between the serverless code execution system 110 and other computing devices. For example, each frontend 120 may include a request interface providing user computing devices 102 with the ability to upload or otherwise communication user-specified code and associated data sets to the on-demand code execution system 110 (e.g., in the form of a disk image) and to thereafter request execution of that code. In one embodiment, the request interface communicates with external computing devices (e.g., user computing devices 102, auxiliary services 106, etc.) via a graphical user interface (GUI), CLI, or API. The frontends 120 process the requests and makes sure that the requests are properly authorized. For example, the frontends 120 may determine whether the user associated with the request is authorized to access the source code specified in the request.
References to source code as used herein may refer to any program code (e.g., a program, routine, subroutine, thread, etc.) written in a specific program language. In the present disclosure, the terms “source code,” “user code,” and “program code,” may be used interchangeably. Source code which has been compiled for execution on a specific device is generally referred to herein as “machine code.” Both “source code” and “machine code” are representations of the same instructions, which may be collectively referred to as “code.” Such code may be executed to achieve a specific function, for example, in connection with a particular web application or mobile application developed by the user. As noted above, individual collections of code (e.g., to achieve a specific function) are referred to herein as “tasks” or “functions,” while specific executions of that code are referred to as “task executions,” “function executions,” “code executions,” or simply “executions.” Source code for a task may be written, by way of non-limiting example, in JavaScript (e.g., node.js), Java, Python, and/or Ruby (and/or another programming language). Tasks may be “triggered” for execution on the serverless code execution system 110 in a variety of manners. In one embodiment, a user or other computing device may transmit a request to execute a task may, which can generally be referred to as “call” to execute of the task (e.g., a “task call,” a “function call,” etc.). Such calls may include an identifier of the task to be executed and one or more arguments to be used for executing the task. A request interface of the frontend 120 may receive calls to execute tasks as Hypertext Transfer Protocol Secure (HTTPS) requests from a user. Also, any information (e.g., headers and parameters) included in the HTTPS request may also be processed and utilized when executing a task. As discussed above, any other protocols, including, for example, HTTP, MQTT, and CoAP, may be used to transfer the message containing a task call to the request interface.
Prior to calling for execution of a task, an end user may define subject code for the task and associated data to be used to execute the task. In one embodiment, the code is provided in the form of a disk image containing the code and other data that the code may use during execution. The disk image and associated metadata for the task (e.g., the end user who “owns” the task or other information regarding the task) may be stored within an object storage system. The object storage system may represent any of a number of object storage systems, such as AMAZON™'s SIMPLE STORAGE SERVICE™ (or “S3™”). In accordance with embodiments of the present disclosure, a disk image may further be divided into a number of portions, each stored as a separate object on the object storage system.
After a user has created a task on the serverless code execution system 110, the system 110 may accept calls to execute that task. To accept calls to execute a task, the frontend 120 can include an execution queue, which can maintain a record of requested task executions. Illustratively, the number of simultaneous task executions by the serverless code execution system 110 is limited, and as such, new task executions initiated at the serverless code execution system 110 (e.g., via an API call, via a call from an executed or executing task, etc.) may be placed on the execution queue and processed, e.g., in a first-in-first-out order. In some embodiments, the on-demand code execution system 110 may include multiple execution queues, such as individual execution queues for each user account. For example, users of the serverless code execution system 110 may desire to limit the rate of task executions on the serverless code execution system 110 (e.g., for cost reasons). Thus, the serverless code execution system 110 may utilize an account-specific execution queue to throttle the rate of simultaneous task executions by a specific user account. In some instances, the serverless code execution system 110 may prioritize task executions, such that task executions of specific accounts or of specified priorities bypass or are prioritized within the execution queue. In other instances, the serverless code execution system 110 may execute tasks immediately or substantially immediately after receiving a call for that task, and thus, the execution queue may be omitted.
As noted above, tasks may be triggered for execution at the serverless code execution system 110 based on explicit calls from user computing devices 102 (e.g., as received at a request interface). Alternatively or additionally, tasks may be triggered for execution at the serverless code execution system 110 based on data retrieved from one or more auxiliary services 106. To facilitate interaction with auxiliary services 106, the frontend 120 can include a polling interface, which operates to poll auxiliary services 106 for data. Illustratively, the polling interface may periodically transmit a request to one or more user-specified auxiliary services 106 to retrieve any newly available data (e.g., social network “posts,” news articles, files, records, etc.), and to determine whether that data corresponds to user-established criteria triggering execution a task on the serverless code execution system 110. Illustratively, criteria for execution of a task may include, but is not limited to, whether new data is available at the auxiliary services 106, the type or content of the data, or timing information corresponding to the data. In some instances, the auxiliary services 106 may function to notify the frontend 120 of the availability of new data, and thus the polling service may be unnecessary with respect to such services.
In addition to tasks executed based on explicit user calls and data from auxiliary services 106, the serverless code execution system 110 may in some instances operate to trigger execution of tasks independently. For example, the serverless code execution system 110 may operate (based on instructions from a user) to trigger execution of a task at each of a number of specified time intervals (e.g., every 10 minutes).
The frontend 120 can further include an output interface configured to output information regarding the execution of tasks on the serverless code execution system 110. Illustratively, the output interface may transmit data regarding task executions (e.g., results of a task, errors related to the task execution, or details of the task execution, such as total time required to complete the execution, total data processed via the execution, etc.) to the user computing devices 102 or to auxiliary services 106, which may include, for example, billing or logging services. The output interface may further enable transmission of data, such as service calls, to auxiliary services 106. For example, the output interface may be utilized during execution of a task to transmit an API request to an external service 106 (e.g., to store data generated during execution of the task).
Code executions triggered on the serverless code execution system 110 of
Each worker 181 may host a number of instances 183A, . . . , 183N. Each of the instances 183A, . . . , 183N may be isolated from other instances 183A, . . . , 183N, thus ensuring the security of code executions on the serverless code execution system 110. For example, each of the instances 183A, . . . , 183N may be divided by a virtualization boundary, by virtue of the instances 183A, . . . , 183N being a virtual machine hosted by the worker 181. In addition, each of the instances 183A, . . . , 183N may exist within a partitioned user space on the worker 181, which logically partitions resources of the worker 181 among the instances 183A, . . . , 183N. Each user space may, for example, represent a “chroot” jail-a known isolation technique for LINUX™ operating systems. Each virtual machine instance may be instantiated with one or more virtualized physical components that are supported by physical components of the host device (e.g., memory, disks, GPUs, etc.). For example, the VM instance 183A may be instantiated with particular virtualized physical components. Each of the instances 183A, . . . , 183N may be dynamically spun up or down based on requests for execution of code modules (e.g., provisioned or released).
To facilitate rapid execution of code, each worker 181 may be configured to maintain a set of instances 183A, . . . , 183N in a “pre-warmed” state, being at least partially configured to begin execution of code. For example, instances may be created on the worker and configured with access to virtualized physical components. In accordance with embodiments of the present disclosure, it may be impractical or impossible to maintain instances 183A, . . . , 183N in a fully warmed state for all possible code executions, as executions may be associated with a wide variety of at least partially distinct data sets (e.g., disk images and/or snapshots). Thus, instances 183A, . . . , 183N may be maintained in a “greatest commonality” for a given group of tasks, such as being provisioned with a set of computing resources common to those tasks, being configured to accept an operating system type used by those tasks, etc.
On receiving instructions to provision an instance 183A to support execution of the task, the worker 181 may adjust the configuration of the instance 183A to support that execution. Specifically, and in accordance with embodiments disclosed herein, the worker 181 may provision the instance 183A with access to a disk image or snapshot corresponding to the task, in a manner that does not require that disk image or snapshot to be fully transferred to local storage of the worker 181 prior to use. Rather, the worker 181 may provide to an instance 183A what appears to be full local access to the disk image or snapshot, while “lazily” retrieving portions of that image or snapshot in response to a request to read such portions. Further, the worker 181 may also provision the instance 183A with access to snapshot of state information by lazily loading the snapshot of the state information. Thus, the worker 181 can provision the instance 183A by lazily loading the disk image and the snapshot of the state information.
The serverless code execution system 110 may load the snapshot of the state information from the snapshot management service 130. The snapshot management service 130 may incrementally or periodically snapshot the state information and store the incremental snapshots or versioned snapshots 135A, . . . , 135N of the state information generated by execution of code modules. The snapshot management service 130 may store the snapshots 135A, . . . , 135N in the snapshot data store 134. Further, the snapshot management service 130 may store version identifiers 133A, . . . , 133N identifying the version of each snapshot in a version data store 132. The snapshot management service 130 may also include a snapshot manager 136 that may store the version identifiers 133A, . . . , 133N and snapshots 135A, . . . , 135N and may identify a particular version identifier and a particular snapshot to the worker 181 in response to a received code module. The snapshot manager 136 may provide the requested incremental snapshots 135A, . . . , 135N to the worker 181 in response to a request to execute a code module. For example, the serverless code execution system 110 may receive a request from the user device 102 to execute a code module. The serverless code execution system 110 may also receive a state information request from the user device 102 for execution of the code module. The serverless code execution system 110 may receive the state information request, and, based on the state information request, the snapshot manager 136 may identify a version identifier corresponding to the state information request. Further, the snapshot manager 136 can identify a particular incremental snapshot corresponding to the version identifier for execution of the code module. The snapshot management service 130 can provide the incremental snapshot to the worker 181 for execution of the code module.
In addition, the system 110 includes a number of components for facilitating distribution of calls to execute a task from frontends 120 to particular VM instances 183A, . . . , 183N. For example, the serverless code execution system 110 includes one or more worker managers 140 configured to manage execution environments (e.g., virtual machine instances) hosted by workers 181 among a worker fleet 180. The worker managers 140—each of which are illustratively implemented as physical or virtual-on-physical devices—illustratively “lease” particular VM instances 183A, . . . , 183N within the fleet 180, thus gaining operational control to, for example, instruct virtual machine instances 183A, . . . , 183N to execute code of the task. Thus, on receiving a call to execute a task, a frontend 120 may distribute the call to a worker manager 140, which may identify a currently-leased VM instance 183A in which to implement the task, and cause the instance 183A to implement the task. Example interactions for distributing a call from a frontend 120 to a worker manager 140 are described, for example, in U.S. patent application Ser. No. 16/698,829, entitled “SERVERLESS CALL DISTRIBUTION TO UTILIZE RESERVED CAPACITY WITHOUT INHIBITING SCALING” and filed Nov. 27, 2019, the entirety of which is hereby incorporated by reference.
In accordance with embodiments of the present disclosure, the snapshot management system 130 may store snapshots 135A, . . . , 135N in a snapshot data store 134 for use in execution of a code module. Further, the snapshot management system 130 may store version identifiers 133A, . . . , 133N identifying the snapshots 135A, . . . , 135N in a version data store 132. The version identifiers 133A, . . . , 133N and the snapshots 135A, . . . , 135N may be managed and stored by a snapshot manager 136.
As illustrated, the device 200 includes a processing unit 290, a network interface 292, a computer readable medium drive 294, and an input/output device interface 296, all of which may communicate with one another by way of a communication bus. The network interface 292 may provide connectivity to one or more networks or computing systems. The processing unit 290 may thus receive information and instructions from other computing systems or services via the network 104. The processing unit 290 may also communicate to and from memory 280 and further provide output information for an optional display (not shown) via the input/output device interface 296. The input/output device interface 296 may also accept input from an optional input device (not shown).
The memory 280 may contain computer program instructions (grouped as units in some embodiments) that the processing unit 290 executes in order to implement one or more aspects of the present disclosure, along with data used to facilitate or support such execution. While shown in
The memory 280 may store an operating system 284 that provides computer program instructions for use by the processing unit 290 in the general administration and operation of the device 200. The memory 280 may further include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, the memory 280 includes a snapshot manager 136 to manage the incremental snapshots as described above. The memory 280 also includes a snapshot generation unit 288 for generation of the snapshots. In some embodiments, the snapshot manager 136 may include the snapshot generation unit 288. In addition, the memory 280 includes the snapshots 286 of state information. The snapshots 286 may be cached locally to the device 200, such as in the form of a memory mapped file.
The snapshot management system 130 of
To further illustrate how the state information is modified or generated by the execution of a code module using a virtual machine instance,
The interactions of
Submission of the code module 103 to the frontend 130 may include transmission of specialized data to the frontend 120, such as a HTTP packet or API call referencing the code module 103 and/or the task to execute the code module 103. While the interactions of
To determine how to execute the code module 103, the serverless code execution system 110, at (2), identifies the code module 103. The serverless code execution system 110 may identify the code module 103 from the received request from the user device 102. The serverless code execution system 110 may also determine whether the request includes a stateful information request (e.g., a request for stateful execution of the code module). If the serverless code execution system 110 determines that the request includes a stateful information request, the serverless code execution system 110 may identify a version identifier corresponding to the stateful information request (e.g., a pointer to the state information). Based on the version identifier, the serverless code execution system 110 may identify a snapshot linked to the version identifier. In the example of
To provide the code module 103 to a particular virtual machine instance, the frontend 120, at (3), distributes instructions to execute the code module 103 to the worker fleet 180. In order to distribute the code module to the worker fleet 180, the frontend 120 can distribute the code module 103 to a worker manager of the worker fleet 180. The frontend 120 may implement various functionalities to distribute the code module 103, such as selecting the worker manager based on random selection, load, etc. In some instances, the frontend 120 may maintain information identifying a worker manager previously associated with a code module, and distribute the code module to that worker manager. Various additional functionalities that may be implemented by a frontend 120 to distribute calls to a worker manager are described, for example, in U.S. patent application Ser. No. 16/698,829, entitled “SERVERLESS CALL DISTRIBUTION TO UTILIZE RESERVED CAPACITY WITHOUT INHIBITING SCALING,” which is hereby incorporated by reference in its entirety. In some embodiments, the worker manager may determine that an appropriate execution environment 302 (e.g., a virtual execution environment or deployment environment such as VM instance 183A or a physical execution environment including one or more physically separated resources) already exists within the worker fleet 180, and may thus execute an instance of the code module within the execution environment 302. In some embodiments, the manager may determine that a new execution environment 302 is required to service the call. Therefore, the worker manager may determine the resources that will be allocated to execute the code module 103. The worker manager may allocate the resources for the execution environment 302 to execute the code module 103. Therefore, the frontend 120 can instruct the worker fleet 180 to execute the code module 103 in an allocated virtual machine instance.
Based on allocating the resources for the execution environment 302, the worker fleet 180, at (4), executes the code module 103 using the execution environment 302. The worker fleet can execute the plurality of code from the code module 103 in the allocated execution environment 103. The execution of the plurality of code may cause modification of state information of components of the allocated execution environment 302. For example, the execution of the plurality of code may modify memory, disks, GPUs, etc. of the allocated execution environment 302. After executing the plurality of code, the worker manager may deallocate (e.g., release) the resources for the execution environment 302. Therefore, the worker fleet 180 executes the code module 103 using an allocated execution environment 302.
To capture the modifications of the state information of the components of the allocated execution environment 302, the snapshot manager 136, at (5), obtains the state information. The snapshot manager 136 may periodically or incrementally obtain state information from the allocated execution environment 302. Further, the snapshot manager 136 may periodically or incrementally obtain the state information while the worker fleet 180 executes the code module 103 using the allocated execution environment 302. For example, the snapshot manager 136 may periodically obtain state information from the allocated execution environment 302. In some embodiments, based on executing the code module, the snapshot manager 136 may obtain the state information from the allocated execution environment 302 prior to the release of the allocated execution environment 302. After the state information is obtained, the allocated execution environment 302 may be released. In some embodiments, the allocated execution environment 302 may not be released.
To store the state information of the components of the allocated execution environment 302, the snapshot manager 136, at (6), generates and stores a snapshot of the state information in a snapshot data store 134. The snapshot manager 136 may generate and store incremental snapshots in the snapshot data store 134. For example, the snapshot manager 136 may generate and store a first snapshot of the state information of the components of the allocated execution environment 302. In some embodiments, the first snapshot may include an indication of the difference between the state information and base state information. In other embodiments, the first snapshot may include an indication of the state information. For a subsequent snapshot, the snapshot manager 136 may receive the updated state information and compare the updated state information with the state information stored in the most recent snapshot (e.g., the first snapshot). Based on this comparison, the snapshot manager 136 can identify the differences between the original state information and the updated state information. The snapshot manager 136 may generate and store a second snapshot of the differences between the original state information and the updated state information. Therefore, the snapshot manager 136 can generate and store the snapshot in the snapshot data store 132.
To identify the state information of the components of the allocated execution environment 302, the snapshot manager 136, at (7), generates and stores a version identifier of the snapshot in a snapshot data store 134. For example, the snapshot manager 136 may generate and store, in the version data store 132, a pointer to the snapshot in the snapshot data store. Further, a plurality of pointers may be made available to a user computing device for selection. For example, a user, via the user computing device, may select a particular pointer (pointing to particular state information) for execution of a set of code) via a user interface that provides information about a plurality of pointers. The snapshot manager 136 can associate each snapshot with a version identifier identifying the version of the snapshot. The snapshot manager 136 may generate and store a plurality of version identifiers in a version data store 132. Each of the version identifiers may be associated with snapshots stored in the snapshot data store 134. Each version identifier may identify a versioned or incremental snapshot generated or modified by the execution of the code module 103. Multiple snapshots and multiple version identifiers may be generated and stored as the code module 103 is executed. The version identifier may identify the code module 103 executed to generate the state information for the snapshot, the allocated execution environment 302, a time of the snapshot, or any other information identifying the snapshot. In some embodiments, the version identifier and the snapshot may be stored in the same data store. In other embodiments, the version identifier and the snapshot may be stored in different data stores. Based on the version identifiers stored in the version data store 132, the snapshot manager may identify particular version identifiers and corresponding snapshots in response to a received stateful information request. Therefore, the snapshot manager 136 may generate and store a version identifier of the snapshot in a snapshot data store 134. In some embodiments, the interactions at (5), (6), and (7) may be carried out iteratively as the code module is executed. Further, the interactions at (5), (6), and (7) may be carried out iteratively as additional code modules are executed.
To further illustrate how subsequent code modules may invoke the previously stored snapshot of the state information,
The interactions of
To determine how to execute the code module 103, the serverless code execution system 110, at (2), identifies the code module 103 associated with a stateful information request (e.g., a pointer to the state information). The serverless code execution system 110 may identify the code module 103 from the received request from the user device 102. The serverless code execution system 110 may also determine that the request includes a stateful information request (e.g., a request for stateful execution of the code module). Further, the serverless code execution system 110 may identify the stateful information request from the request. The stateful information request may identify a version of state information for execution of the code module. Further, the stateful information request may include an alphabetical, numerical, alphanumerical, symbolical, etc. identifier that identifies the version for the execution of the code module. In some embodiments, the serverless code execution system 110 may provide information identifying the versions of the state information to the user device 102 for selection. In order to identify a version of the state information, the request may include information indicating a time of generation of a particular snapshot, a code module executed to generate the particular snapshot, or any other information identifying the version of the state information or the snapshot. Therefore, the serverless code execution system 110 may identify the code module 103 associated with a stateful information request from the request from the user device 102.
To provide the code module 103 to a particular virtual machine instance, the frontend 120, at (3), distributes instructions to execute the code module 103 based on the stateful information request to the worker fleet 180. In order to distribute the code module to the worker fleet 180, the frontend 120 can distribute the code module 103 to a worker manager of the worker fleet 180. Therefore, the worker manager may determine the resources that will be allocated to execute the code module 103. Further, the worker manager may allocate the resources for the execution environment 302 to execute the code module 103. Therefore, the frontend 120 can instruct the worker fleet 180 to execute the code module 103 in an allocated virtual machine instance.
To determine state information for execution of the code module 103, the worker fleet 180, at (4), requests the state information from the snapshot manager 136. The worker fleet 180 may request the state information based on the frontend 120 identifying that the code module is associated with a stateful information request. The worker fleet 180 may provide the stateful information request or a portion of the stateful information request identifying a particular snapshot or set of state information to the snapshot manager 136. Therefore, the worker fleet 180 can request the state information from the snapshot manager 136.
Based on receiving the request for the state information, the snapshot manager 136, at (5), requests and obtains a version identifier of a snapshot. The snapshot manager 136 may request and obtain the version identifier based on the stateful information request from the version data store 132. For example, the stateful information request may include am identifier of a time period of generation of the snapshot or a particular code module (e.g., code module 123) and based on the identifier, the stateful manager 136 can identify a particular version identifier from the version data store 132. The version identifier of the snapshot may identify a particular snapshot in the snapshot data store. The version identifier may include a first subset of information associated with the stateful information request (e.g., a code module identifier) and a second subset of information associated with the snapshot (e.g., a snapshot identifier). For example, the version identifier may include “Code Module Identifier 123” and “Snapshot Identifier 111.” In some embodiments, the version identifier may identify a location of the snapshot. For example, the version identifier may identify a location of the snapshot within the snapshot data store 134. Therefore, the snapshot manager 134 requests and obtain the version identifier of the snapshot. In some embodiments, step (5) may be omitted and the snapshot manager 136 may utilize the version identifier (e.g., the pointer to the state information) to identify the snapshot in the snapshot data store 134.
Based on obtaining the version identifier of the snapshot, the snapshot manager 136, at (6), requests and obtains the snapshot associated with the version identifier. The snapshot manager 136 may request and obtain the snapshot from the snapshot data store 134. The snapshot manager 136 may identify a snapshot associated with the version identifier. For example, the snapshot manager 136 may parse the version identifier to identify the snapshot (or a location of the snapshot) and based on this identification, the snapshot manager 136 can obtain the snapshot from the snapshot data store 134. Therefore, the snapshot manager 136 can request and obtain a snapshot associated with the version identifier.
To enable the execution of the code module based on the state information, the worker fleet 180, at (7), obtains the snapshot of the state information from the snapshot manager 136. In some embodiments, the worker fleet 180 may obtain the state information captured by the snapshot from the snapshot manager 136. Therefore, the worker fleet 180 can obtain the snapshot.
Based on obtaining the snapshot of the state information, the worker fleet 180, at (8), executes the code module 103 using the execution environment 402 based on the state information identified by the snapshot. The worker fleet 180 can execute a plurality of code from the code module 103 in the allocated execution environment 402 based on the state information. The execution of the plurality of code may cause further modification of the state information obtained from the snapshot manager 136. For example, the state information of components of the execution environment may be modified based on execution of the code module. Further, the execution of the code module may modify memory, disks, GPUs, etc. of the allocated execution environment 402. After executing the code module, the worker manager may deallocate (e.g., release) the resources for the execution environment 402 and capture a snapshot of the state information of the components and provide the snapshot to the snapshot manager 136. Therefore, the worker fleet 180 executes the code module 103 using an allocated execution environment 402.
With reference to
In order to execute the code module, at block 504, the serverless code execution system generates a virtual machine instance. The serverless code execution system may generate the virtual machine instance provisioned with the execution environment. In order to generate the virtual machine instance, the serverless code execution system may provision a set of compute resources for the virtual machine for execution of the set of code. In some embodiments, the serverless code execution system may obtain a snapshot of the state information prior to generating the virtual machine instance. For example, the serverless code execution system may determine that the code module is associated with an identifier of state information, obtain the identifier, obtain a snapshot of the state information, and generate a virtual machine instance based on the snapshot of the state information.
In order to determine how to execute the code module, at decision block 506, the serverless code execution system determines if the code module is associated with an identifier of state information. For example, the identifier of state information may be a stateful information request. In providing the code module, a user device may identify a set of code and request that the set of code be executed based on particular state information. In some embodiments, the user device may identify the set of code and the identifier of state information via the code module. In other embodiments, the user device may identify the set of code and the identifier of state information separately. In order to select the identifier of state information, the serverless code execution system may provide the user device with a plurality of identifiers of state information (e.g., identifiers of state information previously generated by the user, related users, users within a particular group, etc.) and the user device may select an identifier of state information from the plurality of identifiers of state information.
If the code module is associated with an identifier of state information, then at block 508, the serverless code execution system obtains the identifier. For example, the code module may reference particular state information and the serverless code execution system may obtain a pointer to a snapshot of the state information based on the code module. In some embodiments, the serverless code execution system may obtain the identifier from the code module. In other embodiments, the serverless code execution system may determine that the code module is associated with an identifier of state information and separately obtain the identifier from a data store. The identifier may identify a particular version of the state information. For example, the identifier may be a version identifier. In some embodiments, the serverless code execution system may determine if the code module is associated with an identifier of state information by obtaining the identifier of state information from the code module (e.g., block 506 and block 508 may be combined).
Based on obtaining the identifier, at block 510, the serverless code execution system obtains the snapshot of the state information. The serverless code execution system may obtain the snapshot of the state information from a plurality of snapshots of the state information. Each of the plurality of snapshots may identify a modified version of the state information.
Based on identifying the snapshot of the state information, at block 512, the serverless code execution system executes the code module based on the snapshot of the state information. In order to execute the code module, the serverless code execution system may initialize the execution environment on the virtual machine instance for execution of the set of code based on the state information. The state information may identify that a component of the execution environment was modified from a first state to a second state. The execution of the set of code may cause the component of the execution environment to be modified from the second state to a third state. Further, execution of the set of code may cause modification of modified state information to previous state information. For example, the component may be modified from the second state to the first state. Further, execution of the set of code may cause modification of a first version of the modified state information and a second version of the modified state information may not be modified. For example, the state information may be forked. Further, execution of the set of code may cause modification of the modified state information based on modifying the state information to generate the modified state information. For example, the modification to generate the modified state information may be redone. Further, execution of the set of code may cause the merging of a first version of the modified state information and a second version of the modified state information. For example, a previously forked set of state information may be merged together and the state information may be combined. Further, the execution of the set of code may cause the selection and/or modification of a portion of the modified state information. Further, the execution of the set of code may cause a filtering of the modified state information. Further, the serverless code execution system may initialize the execution environment based on the state information. For example, the component of the deployment may be configured based on the state information. Further, the state information may identify a state or status of the memory in the execution environment. Based on this state information, the execution environment may be initialized with the component at this state or status. This can enable state information of components to be maintained between executions of code modules.
If the code module is not associated with the identifier of the state information, then at decision block 514, the serverless code execution system executes the code module. The execution of the code modules (with or without the state information) may cause the generation of additional state information. Further, based on executing the code module, the serverless code execution system may incrementally snapshot the component of the execution environment to capture a state of the component. In some embodiments, the component may be a volatile or durable memory of the execution environment and the state information may identify a volatile or durable in-memory state of the execution environment. In some embodiments, based on executing the set of code, the serverless code execution system may release the set of compute resources for the virtual machine. The serverless code execution system may release the set of compute resources prior to execution of a subsequent set of code.
Based on execution of the code module, at block 516, the serverless code execution system generates the snapshot. The serverless code execution system may generate the snapshot based on the execution of the code module. Further, the serverless code execution system may generate a version identifier to identify the snapshot. The serverless code execution system can store the version identifier and/or the snapshot. Further, the serverless code execution system can provide the version identifier and/or the snapshot to a user device. In some embodiments, the serverless code execution system can provide (e.g., cause display of) a plurality of version identifiers and/or snapshots for a user device (e.g., all snapshots associated with a user device, all snapshots associated with a user organization, all snapshots associated with a user role, etc.). Thus, the user device may access a plurality of version identifiers and/or snapshots and select a particular version identifier and/or snapshot for use in the execution of a subsequent set of code. Therefore, the serverless code execution system can enable a user to access snapshots of previous state information in order to select particular state information for the execution of an unrelated set of code (e.g., a set of code that did not generate the state information). In some embodiments, the routine 500 may repeat any number of times. For example, the serverless code execution system may obtain a code module that references a snapshot previously generated at block 516. The routine 500 then ends at block 518.
In various embodiments, the stateful execution routine 500 may include more, fewer, different, or different combinations of blocks than those depicted in
It is to be understood that not necessarily all objects or advantages may be achieved in accordance with any particular embodiment described herein. Thus, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other objects or advantages as may be taught or suggested herein.
All of the processes described herein may be embodied in, and fully automated via, software code modules, including one or more specific computer-executable instructions, that are executed by a computing system. The computing system may include one or more computers or processors. The code modules may be stored in any type of non-transitory computer-readable medium or other computer storage device. Some or all the methods may be embodied in specialized computer hardware.
Many other variations than those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain acts, events, or functions of any of the algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the algorithms). Moreover, in certain embodiments, acts or events can be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines and/or computing systems that can function together.
The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed by a machine, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be a microprocessor, but in the alternative, the processor can be a controller, microcontroller, or state machine, combinations of the same, or the like. A processor can include electrical circuitry configured to process computer-executable instructions. In another embodiment, a processor includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor may also include primarily analog components. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computational engine within an appliance, to name a few.
Conditional language such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, are otherwise understood within the context as used in general to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.
Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.
Any process descriptions, elements or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown, or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.
Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B, and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8782365 | Mooring | Jul 2014 | B1 |
| 8949187 | Satish | Feb 2015 | B1 |
| 9021008 | Andrus | Apr 2015 | B1 |
| 9110600 | Brooker | Aug 2015 | B1 |
| 9189495 | Hughes | Nov 2015 | B1 |
| 9361162 | Higgins | Jun 2016 | B1 |
| 9720806 | Baars | Aug 2017 | B1 |
| 9830345 | Baars | Nov 2017 | B1 |
| 10061613 | Brooker | Aug 2018 | B1 |
| 10108446 | Steinberg | Oct 2018 | B1 |
| 10713080 | Brooker | Jul 2020 | B1 |
| 10776091 | Wagner | Sep 2020 | B1 |
| 10901728 | Negoshian | Jan 2021 | B1 |
| 10963479 | Shah | Mar 2021 | B1 |
| 11042503 | Vig | Jun 2021 | B1 |
| 11099870 | Brooker | Aug 2021 | B1 |
| 11119809 | Brooker | Sep 2021 | B1 |
| 11277494 | McPherson | Mar 2022 | B1 |
| 11487538 | Gove, Jr. | Nov 2022 | B1 |
| 12007943 | Ramachandran | Jun 2024 | B2 |
| 20030093431 | Cooke | May 2003 | A1 |
| 20030101321 | Ohran | May 2003 | A1 |
| 20030158861 | Sawdon | Aug 2003 | A1 |
| 20060101455 | Mantani | May 2006 | A1 |
| 20060184828 | Wynn | Aug 2006 | A1 |
| 20090150872 | Russell | Jun 2009 | A1 |
| 20120209812 | Bezbaruah | Aug 2012 | A1 |
| 20130139128 | Jones | May 2013 | A1 |
| 20130283160 | Ball | Oct 2013 | A1 |
| 20140040888 | Bookman | Feb 2014 | A1 |
| 20140196052 | Arai | Jul 2014 | A1 |
| 20150095885 | Louvau | Apr 2015 | A1 |
| 20150169431 | Ivankovic | Jun 2015 | A1 |
| 20150309829 | Hiltgen | Oct 2015 | A1 |
| 20160110268 | Sekiguchi | Apr 2016 | A1 |
| 20160364440 | Lee | Dec 2016 | A1 |
| 20170085636 | Lin | Mar 2017 | A1 |
| 20170244593 | Rangasamy | Aug 2017 | A1 |
| 20180011895 | Johnson | Jan 2018 | A1 |
| 20180203996 | Gathala | Jul 2018 | A1 |
| 20190155699 | Luo | May 2019 | A1 |
| 20190332267 | Muniswamy-Reddy | Oct 2019 | A1 |
| 20190340109 | Barron-Kraus | Nov 2019 | A1 |
| 20200012619 | Gupta | Jan 2020 | A1 |
| 20200257615 | Richardson | Aug 2020 | A1 |
| 20210117385 | Haldar | Apr 2021 | A1 |
| 20210120039 | Bett | Apr 2021 | A1 |
| 20210240509 | Brooker | Aug 2021 | A1 |
| 20210357297 | Kochar | Nov 2021 | A1 |
| 20210382809 | Lyman | Dec 2021 | A1 |
| 20220019462 | Nishiguchi | Jan 2022 | A1 |
| 20220138048 | Yelheri | May 2022 | A1 |
| 20220317987 | Scrivano | Oct 2022 | A1 |
