Patent Application
20220229768
This disclosure relates to computing systems and related devices and methods, and, more particularly, to a method and apparatus for generating simulated test Input/Output (10) operations.
The following Summary and the Abstract set forth at the end of this document are provided herein to introduce some concepts discussed in the Detailed Description below. The Summary and Abstract sections are not comprehensive and are not intended to delineate the scope of protectable subject matter, which is set forth by the claims presented below.
All examples and features mentioned below can be combined in any technically possible way.
Host applications issue IO (Input/Output) operations on storage systems to access and store data stored on the storage systems. Different host applications will have different IO workload characteristics and, similarly, a given host application may operate differently in different environments. Example applications may include Oracle, SAP, SQL, etc. Example environments may include finance/banking, retail, on-line transaction processing, healthcare, etc. To ensure that a given storage system will meet its service level objectives, it is useful to test the response of the storage system to an expected IO workload.
According to some embodiments, multiple learning processes are trained using live IO operations from different types of reference workloads. Each learning process is trained on a particular reference workload that is generated by a particular application executing in a particular environment. By training each learning process based on live IO operations from an executing application, the learning process is able to learn the IO characteristics of the reference workload issued by the executing application. The IO characteristics, in some embodiments, includes the number of devices used by the application, the range of addresses used by the application, the range of size of IO operations, the sequentiality of the IO operations, and other parameters characterizing the workload characteristics.
Once the learning processes have been trained, parameters describing the trained learning processes, referred to herein as trained model checkpoints, are stored in a reference workload repository along with metadata describing the application and environment where the trained learning process was created. If a storage system is to be tested, the trained model checkpoints of one of the learning processes that was created based on a similar application and environment is selected and loaded to a test server. The test server uses the trained model checkpoints to instantiate a test model configured to generate simulated test IO operations having the same IO characteristics as the reference workload. The test server issues the simulated test IO operations on the storage system under test, to simulate the application workload on the storage system.
Aspects of the inventive concepts will be described as being implemented in a storage system 100 connected to a host computer 102. Such implementations should not be viewed as limiting. Those of ordinary skill in the art will recognize that there are a wide variety of implementations of the inventive concepts in view of the teachings of the present disclosure.
Some aspects, features and implementations described herein may include machines such as computers, electronic components, optical components, and processes such as computer-implemented procedures and steps. It will be apparent to those of ordinary skill in the art that the computer-implemented procedures and steps may be stored as computer-executable instructions on a non-transitory tangible computer-readable medium. Furthermore, it will be understood by those of ordinary skill in the art that the computer-executable instructions may be executed on a variety of tangible processor devices, i.e., physical hardware. For ease of exposition, not every step, device or component that may be part of a computer or data storage system is described herein. Those of ordinary skill in the art will recognize such steps, devices and components in view of the teachings of the present disclosure and the knowledge generally available to those of ordinary skill in the art. The corresponding machines and processes are therefore enabled and within the scope of the disclosure.
The terminology used in this disclosure is intended to be interpreted broadly within the limits of subject matter eligibility. The terms “logical” and “virtual” are used to refer to features that are abstractions of other features, e.g. and without limitation, abstractions of tangible features. The term “physical” is used to refer to tangible features, including but not limited to electronic hardware. For example, multiple virtual computing devices could operate simultaneously on one physical computing device. The term “logic” is used to refer to special purpose physical circuit elements, firmware, and/or software implemented by computer instructions that are stored on a non-transitory tangible computer-readable medium and implemented by multi-purpose tangible processors, and any combinations thereof.
The storage system 100 includes a plurality of compute nodes 1161-1164, possibly including but not limited to storage servers and specially designed compute engines or storage directors for providing data storage services. In some embodiments, pairs of the compute nodes, e.g. (1161-1162) and (1163-1164), are organized as storage engines 1181 and 1182, respectively, for purposes of facilitating failover between compute nodes 116 within storage system 100. In some embodiments, the paired compute nodes 116 of each storage engine 118 are directly interconnected by communication links 120. As used herein, the term “storage engine” will refer to a storage engine, such as storage engines 1181 and 1182, which has a pair of (two independent) compute nodes, e.g. (1161-1162) or (1163-1164). A given storage engine 118 is implemented using a single physical enclosure and provides a logical separation between itself and other storage engines 118 of the storage system 100. A given storage system 100 may include one storage engine 118 or multiple storage engines 118.
Each compute node, 1161, 1162, 1163, 1164, includes processors 122 and a local volatile memory 124. The processors 122 may include a plurality of multi-core processors of one or more types, e.g. including multiple CPUs, GPUs, and combinations thereof. The local volatile memory 124 may include, for example and without limitation, any type of RAM. Each compute node 116 may also include one or more front end adapters 126 for communicating with the host computer 102. Each compute node 1161-1164 may also include one or more back end adapters 128 for communicating with respective associated back end drive arrays 1301-1304, thereby enabling access to managed drives 132.
In some embodiments, managed drives 132 are storage resources dedicated to providing data storage to storage system 100 or are shared between a set of storage systems 100. Managed drives 132 may be implemented using numerous types of memory technologies for example and without limitation any of the SSDs and HDDs mentioned above. In some embodiments the managed drives 132 are implemented using Non-Volatile Memory (NVM) media technologies, such as NAND-based flash, or higher-performing Storage Class Memory (SCM) media technologies such as 3D XPoint and Resistive RAM (ReRAM). Managed drives 132 may be directly connected to the compute nodes 1161-1164, using a PCIe bus or may be connected to the compute nodes 1161-1164, for example, by an InfiniBand (IB) bus or fabric.
In some embodiments, each compute node 116 also includes one or more channel adapters 134 for communicating with other compute nodes 116 directly or via an interconnecting fabric 136. An example interconnecting fabric 136 may be implemented using InfiniBand. Each compute node 116 may allocate a portion or partition of its respective local volatile memory 124 to a virtual shared “global” memory 138 that can be accessed by other compute nodes 116, e.g. via Direct Memory Access (DMA) or Remote Direct Memory Access (RDMA).
The storage system 100 maintains data for the host applications 104 running on the host computer 102. For example, host application 104 may write data of host application 104 to the storage system 100 and read data of host application 104 from the storage system 100 in order to perform various functions. Examples of host applications 104 may include but are not limited to file servers, email servers, block servers, and databases.
Logical storage devices are created and presented to the host application 104 for storage of the host application 104 data. For example, as shown in
The host device 142 is a local (to host computer 102) representation of the production device 140. Multiple host devices 142, associated with different host computers 102, may be local representations of the same production device 140. The host device 142 and the production device 140 are abstraction layers between the managed drives 132 and the host application 104. From the perspective of the host application 104, the host device 142 is a single data storage device having a set of contiguous fixed-size LBAs (logical block addresses) on which data used by the host application 104 resides and can be stored. However, the data used by the host application 104 and the storage resources available for use by the host application 104 may actually be maintained by the compute nodes 1161-1164 at non-contiguous addresses (tracks) on various different managed drives 132 on storage system 100.
In some embodiments, the storage system 100 maintains metadata that indicates, among various things, mappings between the production device 140 and the locations of extents of host application 104 data in the virtual shared global memory 138 and the managed drives 132. In response to an IO (input/output command) 146 from the host application 104 to the host device 142, the hypervisor/OS 112 determines whether the IO 146 can be serviced by accessing the host volatile memory 106. If that is not possible then the IO 146 is sent to one of the compute nodes 116 to be serviced by the storage system 100.
In the case where IO 146 is a read command, the storage system 100 uses metadata to locate the commanded data, e.g. in the virtual shared global memory 138 or on managed drives 132. If the commanded data is not in the virtual shared global memory 138, then the data is temporarily copied into the virtual shared global memory 138 from the managed drives 132 and sent to the host application 104 via one of the compute nodes 1161-1164. In the case where the IO 146 is a write command, in some embodiments the storage system 100 copies a block being written into the virtual shared global memory 138, marks the data as dirty, and creates new metadata that maps the address of the data on the production device 140 to a location to which the block is written on the managed drives 132.
When changes are proposed to be implemented on a particular storage system 100, or a new storage system is to be deployed, it is often desirable to test the storage system to make sure that the storage system will be able to meet desired service level performance metrics. For example, it may be desirable to ensure that the storage system can handle a certain number of IO operations per second while maintaining a response time below a given threshold. To test a storage system, test input/output read/write operations are sent to the storage system, and the behavior of the storage system is monitored to determine whether the storage system is able to meet its service level performance metrics. Input/output operations that are used to test a storage system are referred to herein as “test IO operations”.
Unfortunately, generating test IO operations is not straightforward. Different applications can exhibit different IO characteristics, and accordingly the test IO operations will vary based on the type of application that will be using the storage system for storage of data. Likewise, a given application may exhibit different workload characteristics based on the environment in which it is used. For example, a database application may exhibit different workload characteristics if it is used in connection with on-line transaction processing than it would if used in a healthcare setting.
One way to generate test IO operations is to monitor IO operations of an executing application of the same type in the same environment, and store 10 trace information from the executing application in a memory of a test server. This test IO workload can then be played back by the test server on a storage system under test to determine how the storage system under test responds to the test workload. While this method has the advantage of being based on previous actual application IO traffic, the volume of test IO operations that can be stored is limited by the amount of memory of the test server. Further, due to the size of the 10 test data, it is difficult to use this technique to implement longer time-series tests.
Another way to generate test IO operations is to actually execute the application on the test server, and apply the IOs generated by the application to the storage system under test. Unfortunately, depending on the type of application and expected number of IO operations per second, this can require a very powerful and expensive test server.
According to some embodiments, multiple learning processes are trained using live IO operations from applications in different environments. Each learning process is trained on a particular reference workload that is generated by a particular application that is being used in a particular environment. By training a learning process based on live IO operations from an application operating in a particular environment, the learning process is able to learn the IO characteristics of the reference workload associated with that particular application/environment combination. The IO characteristics, in some embodiments, includes the number of devices used by the application, the range of addresses used by the application, the range of size of IO operations, the sequentiality of the IO operations, and other parameters characterizing the workload of the application in that environment.
Once a learning process has been trained, parameters referred to herein as trained model checkpoints, that describe the learning process, are stored in a reference workload repository along with metadata describing the application and environment where the trained learning process was created.
If a storage system is to be tested, the trained model checkpoints of one of the learning processes that was created based on a similar application and environment is selected and loaded to a test server. The test server uses the trained model checkpoints to instantiate a test model configured to generate simulated test IO operations having the same IO characteristics as the reference workload.
A centralized monitoring system is used to determine overall volume of IO operations—IOPS (IOs Per Second)—of the application executing on other storage systems. This volume information, from the centralized monitoring system, is used to specify the number of IOPS that the test server should generate. The test server issues the simulated test IO operations on the storage system under test, to simulate the application workload on the storage system.
In some embodiments, to generate simulated test IO operations associated with a particular application in a particular environment, a test server obtains the trained model checkpoints of the respective trained model from the reference workload repository. The required devices (storage volumes) are then created on the storage system under test and populated with data. The test server uses the trained model checkpoints to create a test model, and uses the IOPS information from the central monitoring system to generate simulated test IO operations that are applied to the storage system under test. Since the simulated test IO operations are synthetically generated by the test server, the quantity of simulated test IO operations that can be applied to the storage system under test is not limited by test server memory. Since the test server is not required to actually execute the application, but rather simply generates simulated test IO operations intended to mimic the workload of an executing application, an inexpensive server may be used to simply generate simulated test IO operations that have the same workload characteristics as the application would be expected to produce during actual operation.
In some embodiments, the workload monitoring system 200 is implemented using a SLIC (Session Layer Interface Card) that is installed in a storage system 100 that is being used to by host application 104. The workload monitoring system (SLIC) 200 has a GPU (Graphics Processing Unit) 205 and storage 210. The workload monitoring system 200 monitors IO operations by host application 104 as they occur on operating system 150, and uses the IO operations as training examples to train the learning process 220.
By training the learning process 220 online at a customer site based on actual IO operations from an executing application, it is possible to train the learning process 220 without capturing and storing the 10 trace information or transferring the 10 trace information outside of the storage system 100. Further, the learning process 220 can train for a much longer time horizon than would be practical using a trace capture method. Once the learning process 220 is trained, trained model checkpoints 225 describing the trained learning process 220 are transferred to a reference workload repository 300. The reference workload repository 300 stores the trained model checkpoints 225 along with application metadata identifying the type of application and environment in which the application was executing when the learning process 220 was trained.
In some embodiments, the learning process 220 is implemented using a LSTM (Long Short-Term Memory) Neural Network, a RNN (Recurrent Neural Network) or other similar Artificial Intelligence (AI) learning process. The learning process 220 is trained using actual IO operations from an executing application as training examples. In some embodiments, as each IO operation is received, the IO operation is applied to the learning process to adjust the parameters of the learning process based on the characteristics of the IO operation. For example, the size of the IO operation, the address of the IO operation, and other parameters of the IO operation are used, in some embodiments, to adjust the values of nodes of the learning process. Once the learning process reaches a steady state, in which the values of the nodes of the learning process are sufficiently stable, the learning process may be considered to be trained such that the values of the nodes of the learning process describe the characteristics of the reference workload being generated by the application in that environment.
Once the learning process 220 has been trained, trained model checkpoints 225 describing the learning process 220 are transmitted to a reference workload repository 300. The trained model checkpoints 225 are then able to be used to create a test model 405 in a test server 400 (see
Different applications and different environments may generate workloads on a storage system having distinct IO workload characteristics. For example, an application that is used in a retail environment may generate workload on a storage system that has distinct IO characteristics than the same application would generate if it were to be used in a healthcare environment. Likewise, different applications may generate workloads on storage systems that have distinct IO characteristics. As shown in
In some embodiments, the traffic monitoring system shown in
The test server 400 therefore does not need to actually run the application that the storage system will be interacting with once deployed, but rather simply creates a test model 405 to generate a simulated workload having the same workload characteristics as would be expected by the application when executing in the intended environment. Because the test server doesn't need to execute the application, the test server can be implemented using a low-cost computer. Further, because the test server 400 is not storing any actual IO trace data, the size of the memory of the test server does not constrain the amount of simulated test IO operations that the test server 400 can generate. Hence, the test server 400 can continue generating simulated test IO operations indefinitely. Since the test model 405 is based on the reference workload that most closely matches the workload that is to be applied to the storage system 100, the workload on the storage system associated with the simulated test IO operations will mimic the workload that the storage system should expect to encounter when the application is actually deployed in the environment.
For example, assume that a customer would like to deploy a SQL database in a finance/banking environment. To test how the storage system will execute in those conditions, trained model checkpoints 225 obtained from a learning process 220 that was trained in similar conditions will be selected. In this example, trained model checkpoints 225 from a trained learning process 220 will be selected from a learning process that was trained on a storage system where a SQL database was being used in a finance/banking environment. Since the learning process 220 was trained based on a reference workload in these conditions, the trained model checkpoints 225 obtained from that trained learning process 220 will describe the address space and other workload characteristics of this type of workload. The trained model checkpoints 225 are applied by the test server 400 to create and configure the test model 405. The test model 405 then is used by the test server 400 to generate simulated test IO operations on the storage system under test that have the same address space and other workload characteristics.
In some embodiments, training the learning process 220 causes the learning process 220 to learn the address space characteristics of the reference workload. Example address space characteristics might include the number of storage volumes used by the application, the distribution of addresses accessed by the application, the percentage of read and write operations, the size of the read and write operations, the sequentiality of the read and write operations, and other similar parameters that can be used to characterize the reference workload.
As shown in
Since a configured test model 405 is able to recreate the address space characteristics of the reference workload, a test server 400 is able to generate simulated test IO operations that generally simulate a reference workload for a particular application executing in a particular environment. To determine the volume of IOs that should be generated by the test model 405, the centralized monitoring system 500 is used to determine the volume of IOPS to be generated, to enable the test server 400 to mimic production workloads of various applications in a test environment. This enables the workload of multiple environments and applications to be approximated in a laboratory environment using relatively inexpensive test servers 400, since the test servers 400 are simply generating IOs operations to be applied to the storage systems 100, and are not required to store a large number of test IO operations in memory or actually execute any of the applications.
During the training process, parameters of the learning process will vary based on the characteristics of the IO operations. Over time, the parameters of the learning process will approach a steady state, in which the parameters are not changing significantly. At this stage, the learning process has been trained. Accordingly, in some embodiments the learning process is monitored to determine if it has reached a steady state. In particular, in some embodiments the parameters of the learning process are monitored to determine if the parameters have reached a steady state (block 915). If the learning process has not reached a steady state (a determination of NO at block 915), additional IO operations are applied to the learning process to continue the training process. If the learning process has reached a steady state (a determination of YES at block 915), trained model checkpoints 225 describing the trained learning process 220 are transmitted to a reference workload repository 300 (block 920). The trained model checkpoints describing the reference workload are stored in a reference workload repository 300 with application metadata describing the application that generated the reference workload and the environment in which the application was executing (block 925).
The process shown in
As shown in
The test server uses the trained model checkpoints to configure a test model in the test server (block 1015). A set of test devices (storage volumes) is also created on the storage system. In some embodiments, the set of test devices that are created is based on the trained model checkpoints. The devices are also populated with test data (block 1020). For example, if the trained model checkpoints indicate that the test model will generate IO operations on three devices, three devices would be created on the storage system under test. The devices are populated with data to enable the 10 operations to be simulated on the storage system, because the storage system may respond differently if the devices are empty than if the devices contain data.
Once the devices are created on the storage system 100, the test server 400 generates simulated test IO operations on the test devices based on the scaling information and reference model to test how the storage system would be expected to perform if the storage system were to be used by the particular application in that environment (block 1025).
Although some embodiments have been described in which the test server 400 creates a single test model 405 and uses the test model to generate simulated test IO operations, it should be understood that a given test server 400 may create multiple test models 405 and apply generated simulated test IO operations from the multiple test models 405 on the same storage system 100. For example, a given test server may test response of the storage system to multiple applications in multiple environments. Likewise, a given test server may generate and issue simulated test IO operations on multiple storage systems 100.
The methods described herein may be implemented as software configured to be executed in control logic such as contained in a Central Processing Unit (CPU) or Graphics Processing Unit (GPU) of an electronic device such as a computer. In particular, the functions described herein may be implemented as sets of program instructions stored on a non-transitory tangible computer readable storage medium. The program instructions may be implemented utilizing programming techniques known to those of ordinary skill in the art. Program instructions may be stored in a computer readable memory within the computer or loaded onto the computer and executed on computer's microprocessor. However, it will be apparent to a skilled artisan that all logic described herein can be embodied using discrete components, integrated circuitry, programmable logic used in conjunction with a programmable logic device such as a Field Programmable Gate Array (FPGA) or microprocessor, or any other device including any combination thereof. Programmable logic can be fixed temporarily or permanently in a tangible computer readable medium such as random-access memory, a computer memory, a disk drive, or other storage medium. All such embodiments are intended to fall within the scope of the present invention.
Throughout the entirety of the present disclosure, use of the articles “a” or “an” to modify a noun may be understood to be used for convenience and to include one, or more than one of the modified noun, unless otherwise specifically stated.
Elements, components, modules, and/or parts thereof that are described and/or otherwise portrayed through the figures to communicate with, be associated with, and/or be based on, something else, may be understood to so communicate, be associated with, and or be based on in a direct and/or indirect manner, unless otherwise stipulated herein.
Various changes and modifications of the embodiments shown in the drawings and described in the specification may be made within the spirit and scope of the present invention. Accordingly, it is intended that all matter contained in the above description and shown in the accompanying drawings be interpreted in an illustrative and not in a limiting sense. The invention is limited only as defined in the following claims and the equivalents thereto.
