Patent Application
20240202104
The present invention relates to information processing systems. More specifically, embodiments of the invention relate to performing defect prediction operations using information processing systems.
It is known to use information processing systems to perform software testing operations. A software testing operation validates and verifies that a computer program or application functions as expected and/or meets the requirements that guided the design and development of the software application. A software testing operation can be implemented at any time in the development process, but typically occurs after the requirements of the computer program have been defined and the coding process has been completed. Software testing operations often result in the identification of defects in the computer program to allow the defects to be corrected before the computer program is released to users.
A defect describes an error, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes the computer program to behave in unintended ways. Many defects arise from mistakes made in either development of the source code of the computer program or in the design of the computer program. Defects can trigger errors that in turn can have a wide variety of effects with varying levels of inconvenience to the user of the program.
It is known to perform software testing operations via a test suite that when applied to a computer program supposedly ensures that the product functions as expected. However, it has been observed that test suites can be non-exhaustive and that computer program developers generally tend to bypass writing additional tests unless explicitly required to do so.
A system, method, and computer-readable medium are disclosed for predicting a defect within a computer program comprising: accessing a code base of the computer program, the code base of the computer program comprising a plurality of computer program files; training the defect prediction system, the training including performing a historical analysis of defect occurrence patterns in the code base of the computer program; analyzing a commit of the computer program to identify a likelihood of defect occurrence within each of the plurality of files of the computer program; and, calculating a defect prediction metric for each of the plurality of files of the computer program, the defect prediction metric providing an objective measure of defect prediction for each of the plurality of files of the computer program.
The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the several figures designates a like or similar element.
Various aspects of this disclosure include a recognition that an issue relating to computer programs relates to determining when to release a new version of the computer program. It would accordingly be desirable to provide assistance in making this decision in an informed and data-backed way. In various embodiments, the defect prediction operation of the present disclosure provides a methodology where given a commit or a plurality of commits, a prediction may be generated showing for example a number of days before a given file will likely see a defect reported. In various embodiments, the defect prediction operation provides recommendations for corrective actions that can and/or should be taken to mitigate risk associated with the defect predictions.
In certain embodiments, the predictions and recommendations are based on historical analysis of defect occurrence patterns in the code base of the computer program. Various aspects of the disclosure include a recognition that the urgency of the feature or other changes can bias a product owner's judgment on whether to proceed with releasing a new version of the computer program. Accordingly, various aspects of the disclosure include an appreciation that it would be desirable to provide an objective measure of defect prediction.
Various aspects of the disclosed defect prediction operation provides an objective measure of defect prediction. In various embodiments, the objective measure is further refined to provide an objective measure for a particular commit and/or change for a computer program. In various embodiments, the objective measure of defect prediction includes a metric which provides an indication of a predicted interval in which the next defect occurrence can be expected. In various embodiments, the defect prediction operation includes uses an algorithm to leverage a defect occurrence history of a code base and to generate a prediction for next defect occurrence interval. This prediction can be used to prioritize code review activity. The prediction can also be used to determine whether to enforce extensive testing for high-risk files. In certain embodiments, high risk files include those files with smallest relative predicted interval to next defect.
In various embodiments, the defect prediction system further includes a defect prediction user interface. In certain embodiments, the defect prediction user interface presents a dashboard which intuitively provides information to a user regarding files at the greatest risk for defect occurrence. In various embodiments, the dashboard presents information across all file check-ins. Such a dashboard provides a convenient representation for identifying the files that may be desirable to be subjected to additional review and/or testing.
The defect prediction module 118 performs a defect prediction operation. In certain embodiments, the defect prediction operation is performed via a defect prediction system. In various embodiments, the defect prediction operation provides a methodology where given a commit or a plurality of commits, a prediction may be generated showing for example a number of days before a given file will likely see a defect reported. For the purposes of this disclosure, a commit may be defined as a set of computer program files which have completed a commit stage of a software development pipeline. The commit represents a file or set of files of a computer program which represents a release candidate for that file or set of files. In various embodiments, the commit stage can include one or more of a plurality of steps including compiling the code of the file or set of files if necessary, running a set of commit tests on the file or set of files, creating binaries of the file or set of files for use by later stages of the development pipeline, performing analysis of the code of the file or set of files and preparing artifacts, such as test databases, for use by later stages of the development pipeline. In certain embodiments, the defect prediction operation is performed as part of step of performing analysis of the code of the computer program.
In various embodiments, the defect prediction operation provides recommendations for corrective actions that can and/or should be taken to mitigate risk associated with the defect predictions. In certain embodiments, the predictions and recommendations provided by the defect prediction operation are based on historical analysis of defect occurrence patterns in the code base of the computer program. In certain embodiments, the defect prediction operation provides an objective measure of defect prediction. In various embodiments, the objective measure is further refined to provide an objective measure for a particular commit and/or change for a computer program. In various embodiments, the objective measure of defect prediction includes a metric which provides an indication of a predicted interval in which the next defect occurrence can be expected. In various embodiments, the defect prediction operation includes uses an algorithm to leverage a defect occurrence history of a code base and to generate a prediction for next defect occurrence interval. This prediction can be used to prioritize code review activity. The prediction can also be used to determine whether to enforce extensive testing for high-risk files. In certain embodiments, high risk files include those files with smallest relative predicted interval to next defect.
In various embodiments, the defect prediction system further includes a defect prediction user interface. In certain embodiments, the defect prediction user interface presents a dashboard which intuitively provides information to a user regarding files at the greatest risk for defect occurrence. In various embodiments, the dashboard presents information across all file check-ins. In certain embodiments a check-in is a step in the development pipeline where a developer commits a file to a version control system for the computer program. Such a dashboard provides a convenient representation for identifying the files that may be desirable to be subjected to additional review and/or testing.
The defect prediction environment 200 further includes a network 230, which may be a private network, a public network, a local or wide area network, the Internet, combinations of the same, or the like. In various embodiments, the network 140 is included within the network 230. In various embodiment, the defect prediction environment 200 also includes a computer program portion 240. The computer program portion 240 includes a computer program 242 which includes a plurality of files 244. In certain embodiments, the computer program 242 includes the source code for the computer program. In certain embodiments, the computer program portion 240 also includes a version control system 246. Either or both the defect prediction system 210 or the computer program portion 240 may reside within and execute on one or more information processing systems 100. In various embodiments, the defect prediction environment includes a code repository 252 as well as a data repository 254. In various embodiments, the code repository 252 and the data repository 254 store some or all of the information used by the defect prediction system 210 and the computer program portion 240. For example, in certain embodiments, the code repository 252 stores a code base of the computer program. In certain embodiments, the code base of the computer program comprises a plurality of computer program files for the computer program.
As disclosed in greater detail herein, the preparation module 224 performs a preparation phase operation during which an initial risk scored is computed. As disclosed in greater detail herein, the training module 226 performs a training phase operation during which an initial defect prediction model is built. As disclosed in greater detail herein, the verification module 228 performs a verification phase operation during which the defect prediction model is verified.
The user interface system 222 of the defect prediction system 210 provides information to a user regarding the defect prediction. For example, in certain embodiments the user interface provides information regarding defect prediction on a file by file basis. In certain embodiments, the user interface system 222 presents a dashboard which intuitively provides information to a user regarding files at the greatest risk for defect occurrence. In various embodiments, the dashboard presents information across all file check-ins. Such a dashboard provides a convenient representation for identifying the files that may be desirable to be subjected to additional review and/or testing.
In certain embodiments, the computer program 242 comprises the source code for an enterprise software development application. Committing such an enterprise software development application can be the culmination of many months of development with many lines of code written and modified.
In various embodiments, the network 230 may be a public network, such as the Internet, a physical private network, a virtual private network (VPN), a wireless network, or any combination thereof. In certain embodiments, the wireless network may be a personal area network (PAN), based on technologies such as Bluetooth or Ultra Wideband (UWB). In various embodiments, the wireless network may include a wireless local area network (WLAN), based on variations of the IEEE 802.11 specification, often referred to as WiFi. In certain embodiments, the wireless network may include a wireless wide area network (WWAN) based on an industry standard including two and a half generation (2.5G) wireless technologies such as global system for mobile communications (GPRS) and enhanced data rates for GSM evolution (EDGE). In various embodiments, the wireless network may include WWANs based on existing third generation (3G) wireless technologies including universal mobile telecommunications system (UMTS) and wideband code division multiple access (W-CDMA).
In certain embodiments, a defect propensity metric is generated for each of the files 244 of computer program 242. The defect propensity metric provides an objective measure of how risky a file is (i.e., of how likely a file is to have a defect). In certain embodiments, the defect propensity metric is generated as:
Where
In certain embodiments, a repository is a source control system such as a git distributed version control system to which the source code is committed. Effectively, the defect propensity metric for a file is calculated based upon a plurality of criteria. In certain embodiments, the plurality of criteria include: how frequently does the file appear in a defect ridden (i.e., buggy commit) where the higher the number of times a file appears in a buggy commit, the higher the risk is for that file and how recently did the defect ridden commit occur. Thus, the defect propensity metric provides an objective indication which highly corresponds to and confirms an intuitive deduction of a highly skilled computer programmer.
In certain embodiments, the defect prediction system 210 when performing the historical analysis of defect occurrence, the defect prediction system 210 uses a defect classification operation. In certain embodiments, the historical analysis identifies the number of bug-fix commits a particular file has. In certain embodiments, the defect classification operation uses a defect classification tool which generates insights for a particular repository such as classifying the commits into features, defect fixes, configurations, etc.
In certain embodiments, the defect prediction model is based on a plurality of quantifiable factors. More specifically, in certain embodiments, the defect prediction model is based on a risk value (which in certain embodiments may be derived from the defect propensity metric) and a time value representing a time after which a defect occurs in a file having a particular risk value. Based upon an analysis of a large number (e.g., more than 300) open source computer programs, a relationship between the risk value and the time value has been determined. More specifically, referring to
The relation is shown in diagram in
It has been empirically determined based upon an analysis of a large number of software development projects (e.g., more than 300 software development projects), that as the risk metric of a file increases the number of days after which a defect will likely appear in that particular file decreases. I.e., the relation between the risk metric and the number of days after which a defect will likely occur can be represented as an exponential decay. This recognition enables development of a defect prediction model which advantageously provides a metric for predicting future possible defect occurrences relating to a particular file.
An insight derived from this empirical analysis is that for a risk value above a certain amount (in the previous example the certain amount was set as 1250), there exists a relationship exists between two factors. In certain embodiments, the plurality of factors include one or more of the risk and the next defect occurred day. In certain embodiments, the plurality of factors can include a fit value which represents fitting of a higher order polynomial curve to the data and determining a threshold based upon the fitting. This given risk value is referred to as the risk threshold.
Additionally, the risk prediction operation recognizes that the semantics of a risk score can vary from project to project. For example, for a particular project (e.g., Project A) a risk score of 1000 has different consequences (in terms of days to next bug fix) as compared to same risk score of 1000 in project B. Thus, the risk threshold of Project A can be different to that of Project B. Accordingly, the risk prediction operation determines a risk threshold for each project. In certain embodiments, the risk threshold is determined dynamically for every project on which the risk prediction operation is performed.
Referring to
In certain embodiments, whenever a commit occurs, the defect prediction operation 400 calculates a risk metric for each of the files included in the commit. In certain embodiments, the risk metric provides an objective measure for quantifying a prediction of a risk of a file having a defect. In certain embodiments the risk metric represents a predicted amount of time before a defect might occur. In certain embodiments, the predicted amount of time is quantified as a number of days before a defect after which a defect will likely occur in the particular file. In certain embodiments, the risk metric is calculated using a linear regression operation.
During the preparation phase operation 410, the preparation module 224 constructs a commit history to enable computation of computing a risk score. In certain embodiments, a risk threshold value of 20 is selected for this phase. During the training phase operation 420, the training module 226 considers all the commits not present in the preparation phase operation 410 and the verification phase operation 430 for building a first defect prediction model (i.e., Model 1). The training module 226 uses the first defect prediction model to make predictions for commit files during the verification phase operation 430.
More specifically, in certain embodiments the training module 226 performs a linear regression operation to build a linear regression model representing the first defect prediction model.
Referring to
Next, at step 540, a kernel is selected for which the support vector regression operation is to be performed. The kernel which is selected is identified as a Radial Basis function (rbf) kernel. This kernel is used to perform kernelized learning functions such as a support vector machine classification learning function. The kernel corresponds to a similarity of data points in a dataset. Next, at step 550, other parameters of the support vector regression operation are identified. In certain embodiments, the other parameters may include one or more of a penalty parameter, a degree of polynomial kernel function parameters, a kernel coefficient parameter, an epsilon in an epsilon-SVR model parameter, an independent term in kernel function parameter, an indication of whether to use a shrinking heuristic parameter, a tolerance for stopping criterion parameter, a cache size parameter and a hard limit on iterations parameter. In certain embodiments, the other parameters are identified via a parameter estimation operation. In certain embodiments, the parameter estimation operation is based on either or both the penalty parameter and the epsilon in an epsilon-SVR model parameter. In certain embodiments, the optimization operation is performed using a machine learning parameter estimation tool such as the GridSearchCV tool.
Next, at step 560, an accuracy metric is calculated for the selected kernel. Next, at step 565, the accuracy metric is analyzed to determine whether the accuracy metric is below a desired accuracy value and an iteration value is analyzed to determine whether the iteration value is below a predetermined iteration value (indicating that the linear regression operation has performed fewer than a predetermined number of iterations). If both the accuracy metric is below the accuracy value and the iteration value is below the predetermined iteration value, then the dynamic risk threshold is increased by a small amount (e.g., by 2) at step 570, an iteration counter is incremented at step 575 and the linear regression operation 500 returns to step 520. If the accuracy metric is at or above a desired accuracy value or the iteration metric equals the predetermined iteration value then the linear regression operation 500 completes.
In certain embodiments, the dynamic risk threshold is calculated taking into account a plurality of factors. In certain embodiments, the plurality of factors include one or more of the risk and the next defect occurred day and fit. The dynamic risk threshold is then calculated by applying a higher order polynomial curve to the plurality of factors. Afterwards, a risk value is identified for a particular value of “next bug occurred day” from the curve. In certain embodiments, the risk value is set to 30 days, as such a risk value provides a defect prediction for a file in which a defect is likely to occur within a month of the release of the file.
Based on a specific dynamic risk threshold, a build convergence model is generated. The build convergence model is analyzed to determine whether the accuracy of the build convergence model below a specified threshold. In certain embodiments, if the build convergence model is below the specified threshold, the dynamic risk threshold is increased. In certain embodiments, increasing dynamic risk threshold results in an increase in the accuracy of build convergence model. However, in certain embodiments, an increase in the dynamic risk threshold can lead to a less than desirable number (e.g., fewer than 100) of training data points. With this condition, the defect prediction system 210 might generate and a message (e.g., a message setting forth that the defect prediction system has an insufficient amount of data to build a defect prediction model).
In certain embodiments, the defect prediction system 210 determines the accuracy of the defect prediction model based on an upper bound value. In certain embodiments, the accuracy of the defect prediction model is used to indicate whether an actual defect occurs within a file when the defect prediction metric indicated that it would. For example, if the defect prediction metric indicates that a defect will occur in a particular file in a month and an actual defect does occur in the file within a month, then the defect prediction metric is correct and the accuracy of the model using the defect prediction metric is increased. If an actual defect does not occur within the amount of time prescribed by the defect prediction metric then the defect prediction metric is incorrect and the accuracy of the model is decreased.
In certain embodiments, the defect prediction system 210 determines the accuracy of defect prediction model based on an interval way based accuracy calculation. For example, if the defect prediction metric predicted a defect to a file to occur between 15 to 30 days and an actual defect occurs on day 7 then also the prediction is incorrect. In certain embodiments, the accuracy calculation uses an upper bound based accuracy calculation. In certain embodiments, the accuracy value is used internally to further enhance the performance and accuracy of the defect prediction.
In certain embodiments, the verification phase operation 430 uses commits which are present in a predetermined amount of time (e.g., the last 6 months) of the project. The model that is generated during the training operation (e.g., Model 1) is used to make defect predictions on commits in the verification operation for next occurrence of defect. In certain embodiments, the accuracy of the defect prediction model when performing the verification operation is reported via a user interface.
In certain embodiments, the defect prediction system 210 continuously learns and adapts to new changes of computer program code being analyzed. In certain embodiments, after a sufficient number of commits (e.g., 5 commits) are analyzed during the verification phase operation 430, the defect prediction system uses the analyzed commits (commits C1) to as inputs to defect prediction model and a new defect prediction model (Model 2) is generated. The defect prediction system 210 then includes information derived from the analyzed commits when performing a defect prediction operation on commits that are released after the C1 commits (i.e., the new defect prediction model is used when performing the defect prediction operation on commits after the C1 commits).
In certain embodiments, the defect prediction system 210 analyzes the accuracy of the new defect prediction model to determine whether the accuracy of the new defect prediction model is better than the accuracy of the first defect prediction model. If the accuracy of the new defect prediction model is less than the accuracy of the first defect prediction model, then the defect prediction system 210 continues using the first defect prediction model. In certain embodiments, the defect prediction system 210 uses the information used when generating the second defect model when generating a third defect prediction model (Model 3), and so on.
Referring to
Referring to
The present invention may be a system, a method, and/or a computer program product such that selected embodiments include software that performs certain tasks. The software discussed herein may include script, batch, or other executable files. The software may be stored on a machine-readable or computer-readable storage medium, and is otherwise available to direct the operation of the computer system as described herein and claimed below. In one embodiment, the software uses a local or database memory to implement the data transformation and data structures so as to automatically detect library substitution opportunities, improve the quality and robustness of software, educate developers about library opportunities and implementation and generate more readable, reliable, smaller, and robust code with less effort. The local or database memory used for storing firmware or hardware modules in accordance with an embodiment of the invention may also include a semiconductor-based memory, which may be permanently, removably or remotely coupled to a microprocessor system. Other new and various types of computer-readable storage media may be used to store the modules discussed herein. Additionally, those skilled in the art will recognize that the separation of functionality into modules is for illustrative purposes. Alternative embodiments may merge the functionality of multiple software modules into a single module or may impose an alternate decomposition of functionality of modules. For example, a software module for calling sub-modules may be decomposed so that each sub-module performs its function and passes control directly to another sub-module.
In addition, selected aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and/or hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of computer program product embodied in a computer readable storage medium or media having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention. Thus embodied, the disclosed system, a method, and/or a computer program product is operative to improve the design, functionality and performance of software programs by automatically detecting and recommending library function substitutions for replacing validated code snippets in the software program.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a dynamic or static random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a magnetic storage device, a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a Public Switched Circuit Network (PSTN), a packet-based network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a wireless network, or any suitable combination thereof. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Python, Visual Basic.net, Ruby, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language, Hypertext Precursor (PHP), or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server or cluster of servers. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a sub-system, module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
The computer system described above is for purposes of example only, and may be implemented in any type of computer system or programming or processing environment, or in a computer program, alone or in conjunction with hardware. Various embodiments of the present may also be implemented in software stored on a computer-readable medium and executed as a computer program on a general purpose or special purpose computer. For clarity, only those aspects of the system germane to the invention are described, and product details well known in the art are omitted. For the same reason, the computer hardware is not described in further detail. It should thus be understood that the invention is not limited to any specific computer language, program, or computer. The system and method for generating library suggestions may be implemented in any type of computer system or programming or processing environment. It is contemplated that the system and method for generating library function recommendations might be run on a stand-alone computer system, such as the one described above. The system and method for generating library suggestions might also be run from a server computer systems system that can be accessed by a plurality of client computer systems interconnected over an intranet network. Finally, the system and method for generating library suggestions projects may be run from a server computer system that is accessible to clients over the Internet.
Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element of any or all the claims. As used herein, the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17365145 | Jul 2021 | US |
| Child | 18590879 | US | |
| Parent | 15463071 | Mar 2017 | US |
| Child | 17365145 | US |
