Patent Application
20210149805
This disclosure relates to computing systems and related devices and methods, and, more particularly, to a method and apparatus for adjusting cache prefetch policies based on predicted cache pollution from dynamically evolving workloads.
The following Summary, and the Abstract set forth at the end of this application, are provided herein to introduce some concepts discussed in the Detailed Description below. The Summary and Abstract sections are not comprehensive and are not intended to delineate the scope of protectable subject matter, which is set forth by the claims presented below.
All examples and features mentioned below can be combined in any technically possible way.
According to some embodiments, a predictive pollution model is trained to correlate workload sequentiality profiles with pollution levels, under a given pre-fetch policy, and the trained pollution model is then used to determine a prediction of a pollution level on the cache given the sequentiality profile of an observed workload. The predicted pollution level is used to proactively adjust a prefetching policy applied to the cache to attempt to minimize pollution without significantly impacting cache hit rate. In some embodiments, a cache management system includes a sequentiality determination process configured to determine sequentiality profiles of a workload of IO traces as the workload dynamically changes over time. A learning process is trained to learn a correlation between workload sequentiality and cache pollution, and the trained learning process is used to predict cache pollution before the cache starts to experience symptoms of excessive pollution. The predicted pollution value is used by a cache policy adjustment process to change the prefetch policy applied to the cache, to proactively control the manner in which prefetching is used to write data to the cache. Selection of the cache policy is implemented on a per-LUN basis, so that cache performance for each LUN is individually managed by the cache management system.
Data clients 110 act as hosts and provide access to the storage resources provided by storage array 112. In some embodiments, data clients 110 execute in emulations 120 instantiated in the context of the storage system 100. In some embodiments, a hypervisor 122 abstracts the physical resources of the storage system 100 from emulations 120, and allocates physical resources of storage system 100 for use by the emulations 120. Each emulation 120 has an emulation operating system 122 and one or more application processes running in the context of the emulation operating system 122.
Storage array 112 may be implemented using numerous physical drives 113 using different types of memory technologies. In some embodiments the drives 113 used to implement storage array 112 are implemented using Non-Volatile Memory (NVM) media technologies, such as NAND-based flash, or higher-performing Storage Class Memory (SCM) media technologies, such as 3D XPoint and Resistive RAM (ReRAM). Storage array 112 may be directly connected to the other components of the storage system 100 or may be connected to the other components of the storage system 100, for example, by an InfiniBand (IB) bus or fabric.
Storage resources of the storage array 112, in some embodiments, are presented as logical units (LUNs) to the data clients 110 (See
In some embodiments, as shown in
Cache mechanisms are crucial to computer systems such as storage arrays and compute clusters. Correctly placing data with a high probability of being requested on fast memory media can substantially reduce the response times of input/output (I/O) requests.
Resources meant for caching are usually shared among several beneficiaries. Workloads from distinct applications or assigned to different LUNs have different Service Level Agreements (SLAs). Example service levels may include the expected average response time for an IO operation on the LUN, the number of IO operations that may be performed on a given LUN, and other similar parameters.
Cache performance may be measured by looking at what percentage of reads are able to be serviced from the cache. If an address is requested by an application such as data client 110, and the address is contained in the cache 118, a cache hit occurs and the read request can be serviced from the cache 118. If an address is requested by an application such as data client 110, and the address is not contained in the cache 118, a cache miss occurs and the read request must be serviced from the discs 113 of the storage array 112. The percentage of cache hits is referred to herein as cache hit ratio.
There is no general rule that specifies the size of a cache required to obtain a given cache hit ratio. The intrinsic dynamics of the Least Recently Used (LRU) eviction policy and the stochastic behavior of the workload makes the functioning of the cache difficult to predict.
It is possible to prefetch data into the cache in an attempt to increase the hit rate associated with the cache 118. Prefetching is a well-known technique to optimize cache behavior. It exploits data locality with the assumption that applications often request data residing in sequential addresses in a given address space. Namely, when a device's operating system receives a data access request at a certain address, Ai, the system retrieves not only the content of that location, but also the content of the N subsequent address locations, {Ai+1, . . . , Ai+N}, before the application actually requests the data from those other addresses. The operating system then places the retrieved content in the cache 118, which, by design, is a much faster media than the discs 113 of storage array 112 where the data originally resides. If the application (e.g. data client 110) indeed requests data from subsequent addresses, the operating system satisfies those requests directly from the cache 118, instead of fetching data from the slower media at each request.
Unfortunately, cache performance tends to be affected as the workload dynamically changes over time. The frequency of requests, their sizes, and how sequential the reads are can change over time, and these changes can drastically affect how much benefit a particular cache policy will provide. For example, if a workload changes from initially having primarily sequential requests, and then changes to having primarily random requests, using the same cache policy under these different workload characteristics may cause cache performance to vary dramatically.
Storage systems typically have a single prefetching policy, with a cache policy that applies to the entire cache and has a single fixed prefetch look-ahead window. This is not ideal because several applications, with different data access patterns, may access the storage system resources concurrently, and each access pattern traverses the address space of the system distinctly. For instance, some workloads might be sequential, while other workloads might be random; some workloads might traverse the entire address space, while other workloads might be concentrated in a small range of addresses. A given prefetch policy may work quite well with one workload access pattern, while generating excessive pollution when used with another workload access pattern.
One way to determine whether a cache policy is correctly being applied to the cache 118 is to measure cache performance by simply calculating cache hits, misses, and pollution levels. However, such measurements only indicate how the cache is responding to the workload characteristics. Alone, these measurements/indicators cannot explain why a cache policy is not performing well at any given point in time, or predict how changes to the cache policy are likely to affect future cache behavior.
According to some embodiments, a mechanism is described that automatically and dynamically enables and disables a prefetch cache policy on a per LUN basis, depending on the predicted pollution anticipated by application of the cache policy to the cache, given a sequentiality profile of the current workload. In some embodiments, sequentiality profiles of the workloads are leveraged to predict pollution and for decision making. Experimental results demonstrate that dynamically enabling and disabling prefetching can significantly reduce cache pollution and I/O overhead while causing low impact on cache hit rates.
The biggest challenge to decide when and how much data to prefetch is the fact that I/O workloads and data access patterns are dynamic, such that they vary over time. Wrong assumptions about the sequentiality of the data at different points in time may lead to data being prefetched without ever being requested by applications. This problem is referred to herein as cache pollution. In situations where excessive cache pollution occurs, prefetching may cause severe performance degradation. Cache pollution is detrimental to system performance because it needlessly increases data traffic due excessive data fetching, potentially causing bottlenecks in the memory bus. Cache pollution also accelerates the cycling of data in the cache, which can cause the premature eviction of content that could be requested by the applications, potentially leading to more cache misses and more data fetching.
According to some embodiments, a method of automatically switching prefetching on and off is provided, which is based on a prediction of how much pollution will occur in the cache in a subsequent time period based on a current sequentiality profile of the workload on the cache in the current time period. Whenever the predicted pollution is above a certain threshold, prefetching is switched off. Conversely, prefetching is switched back on when the predicted pollution is below the threshold. By doing this, the system is able to achieve high hit ratios by turning on prefetching when the workload exhibits primarily sequential access patterns, and the system is able to reduce pollution by turning off prefetching when the workload exhibits primarily random access patterns. By implementing this policy determination and adjustment process periodically, the cache management system is able to prevent pollution from building up in the cache, and is able to implement cache management on a per-LUN basis.
Turning prefetching on or off does not affect previous workload and only affects cache performance in connection with subsequent requests. Accordingly, to proactively determine whether prefetching should be enabled/disabled, it is necessary to characterize the nature of expected future requests. Thus, predicting disk access patterns is at the core of the prefetching problem. The challenge includes determining when and if an applications' upcoming I/O requests will be in the range of the prefetched requests.
Intrinsically, prefetching is beneficial over sequential access patterns. Indeed, many applications and file types can be associated with sequentiality of access. For instance, image, audio and video processing applications generally process file content sequentially. However, from the perspective of the storage system 100, files are just collections of bytes, and applications are only seen through the lens of the I/O requests they issue to the storage system 100. The cache management system 128 does not have visibility to the type of application that is issuing the IO requests or to the type of data that is being accessed via the IO request. As a result, the challenge for the cache management system 128 is to try to predict future access patterns and prefetching performance, without any knowledge about the nature of the applications and file types.
The pollution levels caused by different prefetching policies can vary greatly under the same I/O workload. For example, one prefetching policy may be to pre-fetch four blocks of data each time an IO request is received, whereas another prefetching policy may be to pre-fetch eight blocks of data each time an IO request is received. Yet another example policy may be to pre-fetch eight blocks of data when two subsequent IO requests are received. Thus, it is necessary to consider the problems described above in a policy-aware manner.
Any predictive mechanism is subject to prediction errors. While errors might be tolerable to a certain extent in some instances, erroneously switching a prefetching policy on and off may lead to unacceptable costs in terms of resource management. Therefore, the dynamic approach relying on prediction must also be robust with regards to errors in the estimation or prediction of the pollution levels in future data access. As shown in
The predictive pollution value provided by the learning process, in some embodiments, is used by a cache policy adjustment process 752 to change the prefetch policy applied to the cache to proactively control the manner in which prefetching is used to write data to the cache in dependence on the dynamically changing workload. The cache management system 128, in some embodiments, implements the method of predicting cache performance for each LUN to individually manage whether prefetching is turned on or turned off for each LUN based on the current sequentiality profile of the workload associated with that LUN.
According to some embodiments, a predictive pollution model is trained to correlate workload sequentiality profiles with pollution levels, under a given pre-fetch policy, and the trained pollution model is then used to determine a prediction of a pollution level on the cache given the sequentiality profile of an observed workload. The predicted pollution level is used to proactively adjust a prefetching policy applied to the cache to attempt to minimize pollution without significantly impacting cache hit rate.
Although some embodiments will be described in which the learning process is trained to learn a correlation between the sequentiality of workloads and cache pollution levels, it should be noted that other mechanisms for predicting the cache pollution caused by a prefetching policy could be used instead, and the invention is not limited to an implementation that relies on predicting the sequentiality of subsequent workloads to dynamically adjust the prefetching policy applied to the cache.
Determining sequentiality relies on the existence of I/O telemetry data that indicates how applications traverse the address space of a storage system 100. As shown in
In some embodiments, the cache management system 128 does not know the type of application that generated the I/O, but rather only has access to storage telemetry data in the form of I/O traces. An IO trace, as that term is used herein, is a collection of pieces of information associated with an IO operation that indicates what type of I/O operation the application issued (e.g., ‘read’ or ‘write’), the size of the operation, a timestamp associated with the operation, and in indication of an address in the LUN's addressable space. An example of such storage telemetry data is shown below in Table I. In Table I:
The example storage telemetry data, in the form of IO traces shown in Table I, is able to be collected in a storage system 100 and, as discussed in greater detail herein, can be exploited by sequentiality determination process 751 to determine how I/O workloads and access patterns vary with time.
In the following discussion, the logical addressable space, such as the logical address space of the LUNs shown in
One of the core challenges in the analysis of and reasoning based on I/O traces is the large number of requests processed in small periods of time. Processing large amounts of requests in efficient manners is required for any kind of online decision making. According to some embodiments, sequentiality is determined using a set of data structures described below in connection with
In some embodiments, the sequentiality determination process 751 implements computes a histogram of sequence lengths every time an IO occurs on a given LUN. In some embodiments, the sequentiality determination process 751 uses the set of data structure shown in
According to some embodiments, the sequentiality determination process 751 first finds sequences in the telemetry data comprising collections of address related to subsequent I/O requests that were sent to the same LUN. Next, those patterns are used to compute a sequentiality histogram for the LUN, specifying how many sequences of each length occurred in the set of traces being considered. An example histogram is shown in
The objective of creating a sequentiality profile of the workload is to collect the lengths of the sequences formed in this sorted segment of addresses.
From the ordered sequence of traces, a list of sequences data structure 620 is created. The list of sequences data structure 620, in some embodiments, is implemented as a double linked list. Each sequence is an element of the list of sequences data structure 620, and each element points to the previous sequence and subsequent sequence.
In some embodiments, a sequence lengths data structure 630 is used to summarize the lengths of the sequences contained in the list of sequences data structure 620, and a histogram data structure 640 correlating sequence lengths with the number of sequences of each length is created from the sequence lengths data structure 630.
In some embodiments, a method is defined to reduce the cache pollution of systems that make use of pre-fetching. As used herein, prefetch pollution (P) is defined as the number of pages that were prefetched into the cache 118 and were never used. The pollution ratio (Rp), corresponding to a given window of requests, is defined as:
where Qp is the total number of prefetched pages in the period under consideration.
Thus, Rp values range between 0 and 100, representing the percentage of prefetched pages that are never used before being evicted from the cache. An Rp value of 0% corresponds to all prefetched pages being used (perfect prefetching), an Rp value of 100% corresponds to no pre-fetched pages ever being used. Ideally, it is preferable to enable prefetching only when beneficial, and to disable it otherwise.
In some embodiments, the computed sequentiality profiles described above are used to predict a pollution ratio, which is used to turn on/off prefetching. This embodiment corresponds to the implementation that was used in the experimental validation discussed below in connection with
As shown in
Features are then extracted from the data over a predetermined period, t (
The features (e.g. sequentiality) extracted from the trace data of the workload are provided as input to a predictive pollution model 750, yielding a predicted pollution ratio . The method then proceeds to determine, based on the predicted pollution ratio
, whether prefetching should be turned on or off (
with a determined threshold pollution ratio Rcut. If the pollution ratio estimate
is below the determined threshold pollution ratio Rcut, (a determination of NO at block 800), pollution is predicted to be relatively low for the upcoming time period if prefetching is enabled and, accordingly, the cache policy adjustment process 752 turns prefetching ON for that LUN for the following period t (
is above or equal to the determined threshold pollution ratio Rcut, (a determination of YES at block 800), pollution is predicted to be relatively high for the upcoming time period if prefetching is enabled and, accordingly, the cache policy adjustment process 752 turns prefetching OFF for that LUN for the following period t (
The selected prefetch policy is then applied to the cache 118 by the cache policy adjustment process 752 of cache management system 128 for a predetermined period t (
In some embodiments, the period t is defined to be one minute. This is based on empirical observations which indicated that this is a long-enough period to characterize a state of the workload and benefit from cache configuration updates, including those related to model retraining and deployment. Other time periods t may likewise be used, and the invention is not limited to an implementation in which the cache management system 128 adjusts the cache prefetch policies on a per LUN basis once each minute.
In some embodiments, a predictive pollution model 750 is trained with historical trace data. The task of the training step (
In some embodiments, the predictive pollution model 750 is a learning process implemented using a regression model. Regression analysis is a form of predictive modelling technique which investigates the relationship between a dependent variable y (target) and independent variable x (predictor). As shown in
There are several types of regression analysis that can be used for forecasting, time series modelling, and finding the causal effect relationship between the variables. In some embodiments, the predictive pollution model is implemented using a regression model trained to predict the future pollution ratios incurred by a prefetching policy under a certain workload profile. In some embodiments, the regression model is a support vector regression (SVR) learning process. In some embodiments, the regression model is a regression tree learning process. Other regression models may be used as well depending on the implementation.
SVR typically produces results more accurate than regression trees, at the expense of being more computationally costly and less interpretable. The regression tree, on the other hand, is interpretable, which is beneficial in that it is possible to determine why prefetching was enabled or disabled at a given time window. Specifically, with a regression tree model, given a sequentiality profile set as input to a decision tree, it is possible to determine which features are used to predict the value of the metric of interest, e.g., pollution.
Random I/O patterns diminish the benefits of prefetching and may incur high levels of pollution. In those cases, the method should correctly predict an increased level of pollution and pro-actively disable prefetching to prevent performance degradation. Conversely, in sequential I/O patterns, the method should predict a reduction in pollution levels (assuming the prefetching policy correctly exploits the sequential pattern) and switch the prefetching policy back on again.
In some embodiments, the predictive pollution model 750 is trained with sequentiality profiles, such as the histograms described above and shown in
Once the predictive pollution model 750 is trained, it is deployed (
Thus, in some embodiments, the predictive pollution model 750 learns a relationship between workload sequentiality and cache pollution, and uses that learned relationship to identify and predict the levels of cache pollution expected to be generated by a workload with a certain sequentiality profile and under a certain prefetching policy. Then, that prediction is then used by the cache policy adjustment process 752 of the cache management system 128 to turn prefetching ON or OFF for the subsequent processing period t. If, based on the currently observed sequentiality profile, the predictive pollution model 750 predicts that pollution for the subsequent period is expected to be above a certain threshold, cache prefetching is switched off. Otherwise, it remains active. As discussed below, the threshold may be determined in multiple ways, but in some embodiments is determined in part by the available throughput in the memory bus and by the cache size.
In some embodiments, the predictive pollution model 750 is trained to predict using feature array 770 containing training data.
In some embodiments, the sequentiality profile 1005 is computed, as described above, as a histogram computed from dynamically changing workload of I/O requests on the storage system. The measured cache pollution levels 1010 that occurred in the cache 118 during the period of time associated with the sequentiality profile is obtained by observing the actual operation characteristics of the cache 118 while processing the IO requests. The cache prefetch policy 1015, in some embodiments, is a binary value indicating whether the cache prefetch policy was turned ON during the time interval or whether the cache prefetch policy was turned OFF during the interval.
The entries of training data contained in feature array 770 may be based on actual historical data experienced by a storage system, collected over time intervals, or may be based on simulations of cache behavior from previously collected trace data. In some embodiments, each time interval such as each minute or some other interval of different length, the system computes the sequentiality profile 1005 for the workload of IO traces and collects metadata including the prefetch policy and measured pollution ratio Rp 1010. In some embodiments, a significant amount of training data is collected, such as by collecting training data on a storage system over a period of days or weeks, to enable the feature array 770 to have a significant number of entries 1000.
In an environment where data remains in the cache for a period (eviction period) that significantly exceeds the time period used by the predictive process shown in
Once a set of training data has been collected, the training data is applied to train the predictive pollution model 750 (
In some embodiments, to evaluate the quality of the trained model, the training data is split such that some of the data of the feature array 770 is used to train the predictive pollution model 750, and the other portion of the data of the feature array 770 is used to test the predictive pollution model 750. For example, 60% of the data of the feature array 770 might be used to train the model and the other 40% of the data of the feature array 770 might be used to test the accuracy of the trained predictive pollution model 750. Experimentally, it was determined that the two regression algorithms, SVR and regression tree, empirically performed equally well, considering the R2 quality measure.
Notice that policies such as SegDeltaLRU8 try to detect patterns in sequences of requests, but they do not always succeed. By leveraging the predictor, it is possible to anticipate when a prefetch policy such as the SegDeltaLRU8 policy will fail. , from the trained predictive pollution model 750. In this example, the mean squared error of the predictions (MSE) was 0.03, the mean absolute error (MAE) was 0.13, and the R2 was 0.48.
As shown in this experiment, it is possible to empirically demonstrate that the process described herein is effective in reducing the pollution of a caching policy that uses prefetching, with a small impact on the hit ratios. The main practical application of the method occurs when a cache system is either (a) diagnosed with response time problems due to prefetching or (b) is likely to suffer from performance degradation due to congested memory bus or small cache size. In those cases, it is possible to apply the method to reduce or prevent the increase of latency, by disabling prefetching when pollution is predicted and leaving the prefetch activated when the pollution is low.
Referring back to
In some embodiments, using the trained predictive pollution model, the system predicts the at every time period t (e.g. each minute) for a given cache policy. If the
is greater than a given threshold value, referred to herein as to as Rcut, prefetching is disabled (a determination of YES at block 800). Otherwise, prefetching is enabled (a determination of NO at block 800).
The particular value selected for Rcut may depend on the particular storage system. There are several aspects involved in the tuning of Rcut. Experimentally, the value of Rcut was varied to assess its impact on the metrics of interest. It was found that the allowed cache pollution grows as Rcut increases. The dependency of hit ratios on Rcut, in contrast, is not straightforward. It was experimentally observed that as Rcut increases, hit ratios also increase, although the increase in hit ratios is usually less significant than the growth in pollution.
In some embodiments, the particular value of Rcut depends on (1) characteristics of the disk access pattern; (2) cache size; and (3) the maximum throughput supported by the memory bus.
If the disk access pattern is mostly non-sequential, there is little benefit in prefetching regardless of the cache size and memory bus throughput, and Rcut should be set to a small value (to keep prefetching disabled most of the time). In contrast, if the cache size and memory bus throughput are very large, and the I/O access pattern is substantially sequential, Rcut should be set to a large value (to enable more aggressive prefetching) over a larger number of sequentiality profiles.
In the situation where the cache size and/or memory bus throughput are limited, the value of Rcut should be heuristically (self) determined and a more detailed analysis needs to be used to enable the particular storage system 100 to determine the appropriate value of Rcut that will result in optimal results for that particular storage system.
In some embodiments, enabling the storage system 100 to heuristically determine the appropriate value for Rcut may include several tests. In some embodiments, if there is unlimited memory bus throughput, the value of Rcut is initialized as 0%, and then increased as the hit rate increases. As soon as the hit rate stops increasing, a candidate value for Rcut has been determined.
In some embodiments, if there is unlimited cache size, the value of Rcut is initialized as 0%, and then increased while memory bus throughput is not saturated, i.e., while system response times remain generally constant. As soon as response times increase, a candidate value for Rcut has been determined.
In some embodiments, if there is both a limited cache size and limited memory bus throughput: Rcut is initialized as 0%, and then increased while memory bus throughput is not saturated and hit ratios increase. Whenever one of those conditions is not met, a candidate value for Rcut has been determined.
However, by introducing the threshold Rcut into the system, the number of errors by the predictive pollution model that affect operation of the cache management system are significantly decreased. To illustrate this,
The two situations where an improper pollution prediction by the predictive pollution model 750 will make a difference are where the predictive pollution model 750 predicts a pollution level above Rcut, and the actual pollution is below Rcut (quadrant 1300) or where the predictive pollution model 750 predicts a pollution level below Rcut, and the actual pollution is above Rcut (quadrant 1310). Specifically, these types of errors may affect performance of the cache, because the cache management system will incorrectly turn off prefetching where the predictive pollution model 750 predicts a pollution level above Rcut, when the actual pollution is below Rcut (quadrant 1300). Conversely, the cache management will incorrectly turn on prefetching where the pollution model 750 predicts a pollution level below Rcut, when the actual pollution is above Rcut (quadrant 1310). Accordingly, as shown in
For example, as shown in
Note that the configuration of the method shown in
Experiments were performed using these described methods using real-world workload trace data from a particular storage system. In particular, one goal was to illustrate the impact of Rcut on the metrics of interest, and to illustrate that there is an elbow on the value of Rcut such that if Rcut is set to that value the hit rate is marginally affected whereas pollution significantly decreases.
The input data for the experiments was a data access trace workload corresponding to 1400 minutes of disk accesses from 6 different machines. In those machines, 760 requests were made per second, on average. As a result, the experimental dataset included roughly 45,600 requests per minute. For every minute of workload, a corresponding sequentiality profile was generated. Thus, there were a total of 1400 samples of sequentiality profiles. Those samples were randomly separated into two subsets: a subset with 1000 samples was used to create and train the prediction models, and the additional 400 test samples were used to generate the results shown in
For the cache simulations of each system present in the workload, a cache size of 1% of the maximal address in the data access trace workload was considered, and it was assumed that each request occupied exactly one cache position. The prefetching policy employed was the SegDeltaLRU8 referenced above. Thus, the proposed method switched from SegDeltaLRU8 policy to Least Recently Used (LRU) cache eviction policy without prefetching, every time the predicted pollution was above Rcut, and switched back to SegDeltaLRU8 when the predicted pollution was below Rcut. Varying the Rcut parameter, resulted in the values shown in the Table II:
It is apparent that the tradeoffs between hits and pollution yielded by varying values of Rcut are significant. If the impact of pollution in a particular system is too significant, a relative pollution value of 30% might be too high. In this case, an optimal value of Rcut may fall in the 25%-50% range. Thus, as described above, in some embodiments it is advantageous to determine the optimal Rcut values with knowledge of the resource constraints in the actual domain. In any case, the experiments also demonstrated the benefits of using a method to predict pollution levels from sequentiality profiles and to make decisions about policy switching based on such predictions.
The methods described herein may be implemented as software configured to be executed in control logic such as contained in a Central Processing Unit (CPU) or Graphics Processing Unit (GPU) of an electronic device such as a computer. In particular, the functions described herein may be implemented as sets of program instructions stored on a non-transitory tangible computer readable storage medium. The program instructions may be implemented utilizing programming techniques known to those of ordinary skill in the art. Program instructions may be stored in a computer readable memory within the computer or loaded onto the computer and executed on computer's microprocessor. However, it will be apparent to a skilled artisan that all logic described herein can be embodied using discrete components, integrated circuitry, programmable logic used in conjunction with a programmable logic device such as a Field Programmable Gate Array (FPGA) or microprocessor, or any other device including any combination thereof. Programmable logic can be fixed temporarily or permanently in a tangible computer readable medium such as random-access memory, a computer memory, a disk, or other storage medium. All such embodiments are intended to fall within the scope of the present invention.
Throughout the entirety of the present disclosure, use of the articles “a” or “an” to modify a noun may be understood to be used for convenience and to include one, or more than one of the modified noun, unless otherwise specifically stated.
Elements, components, modules, and/or parts thereof that are described and/or otherwise portrayed through the figures to communicate with, be associated with, and/or be based on, something else, may be understood to so communicate, be associated with, and or be based on in a direct and/or indirect manner, unless otherwise stipulated herein.
Various changes and modifications of the embodiments shown in the drawings and described in the specification may be made within the spirit and scope of the present invention. Accordingly, it is intended that all matter contained in the above description and shown in the accompanying drawings be interpreted in an illustrative and not in a limiting sense. The invention is limited only as defined in the following claims and the equivalents thereto.
