Patent Grant
11567824
One or more aspects relate, in general, to information technology systems, and in particular, to cognitive test environments for identifying and resolving failures of information technology systems.
An extensive test process phase is performed during the manufacturing of information technology systems and/or at other times to ensure that the systems will perform as expected when in a production environment. The testing ensures the reliability, accessibility, functionality and/or security of a system.
Some failures during testing are trivial, while others are extensive requiring complex analysis, which takes time and slows down the test process. This inevitably impacts availability of the system in production.
Shortcomings of the prior art are overcome, and additional advantages are provided through the provision of a computer-implemented method. The computer-implemented method includes obtaining, by a testing system based on a system failure, an indication of one or more tasks to be performed. A determination is automatically made, by the testing system, of whether a user is to perform one or more actions to alter execution of the one or more tasks. The automatically determining is based on an indication of whether a knowledge transfer score of the user meets at least one predefined criterion. Based on the knowledge transfer score meeting the at least one predefined criterion, the one or more actions are performed.
Computer systems and computer program products relating to one or more aspects are also described and claimed herein. Further, services relating to one or more aspects are also described and may be claimed herein.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
One or more aspects of the present invention are particularly pointed out and distinctly claimed as examples in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of aspects of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
In accordance with one or more aspects of the present invention, a test advisor facility is provided that performs cognitive test advisor facility processing based on a failure of a system, such as an information technology system being manufactured. The cognitive test advisor facility processing restricts, in accordance with an aspect of the present invention, a selected input in the processing allowing the facility to control the processing, increasing performance and potentially reducing errors. In one example, the selected input is a user's input, including knowledge transfer, with respect to one or more aspects of recovery from a failure. The user's input is restricted, for instance, as the cognitive test advisor facility gains knowledge relating to the failure or type of failure. As an example, a knowledge transfer score is determined, and if the knowledge transfer score has a predefined relationship (e.g., less than or equal) to a threshold, then the user's input is taken into consideration. For instance, one or more actions are performed, such as modifying an activity execution sequence of a plurality of tasks and/or altering selection of a task performing agent. Various other actions may also be performed, either in addition to and/or in lieu of, alteration of the activity execution sequence and/or the task performing agent selection.
In one or more aspects, a dynamic decision-making and learning technique is provided that is based on a user's knowledge transfer for, e.g., the activity sequencing and/or performing agent selection by adoption of the user's cognitive experience. In one embodiment, for a failure-action pair recommendation proposed by a machine learning algorithm, one or more aspects have the ability to restrict on the cognitive learning based on the knowledge transfer score and knowledge transfer threshold indices. The knowledge transfer score is a determining factor on the ability of the computer-based system to autonomously perform one or more actions and/or alter execution of one or more actions without any user (e.g., human) intervention.
Throughout this description, the terms test advisor facility, facility, test advisor tool, test advisor, test advisor system, testing system, cognitive test advisor facility and cognitive machine learning tool and system are used interchangeably to refer to certain aspects of the present invention that perform various operations, functions, and the like. As the terms are used herein, the test advisor facility, tool and system can be an implementation of the functions or aspects of the illustrative embodiments in the form of an apparatus, a procedure, or a computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatuses, computers, data processing systems, or the like. In the case of a computer program product, the logic represented by computer code or instructions embodied in or on the computer program product is executed by one or more hardware devices in order to implement the functionality to perform the operations associated with the specific facility. Thus, the facility described herein can be implemented as specialized hardware, software executing on general purpose hardware, software instructions stored on a medium such that the instructions are readily executable by specialized or general purpose hardware, a procedure or method for executing the functions, etc., or a combination of any of the above.
In one or more aspects, the cognitive test advisor facility analyzes structured and unstructured data to more quickly determine possible test failure repair actions. In one embodiment, the facility performs data analytics on previously run tests, planned tests, failure information, and user feedback data to determine one or more best-possible repair actions to take, based on a particular test failure.
One or more embodiments provide a test advisor facility that resolves manufacturing systems test failures by analyzing test failures and recommending a next best action(s) via one or more machine learning processes. In some embodiments, this includes the ability to identify and propose best possible test repair actions based on historical data and cognitive learning capability relating to a particular detected quality problem. Cognitive machine learning can be used to identify root causes or symptoms over time for each test failure. In this manner, similar test failures can be automatically assessed and resolved based on previous test results and on feedback stored in a data repository. In some embodiments, several variables can be accounted for in the allocation of a repair part when resolving a particular problem. In some embodiments, the variables include part availability, the cycle time for the repair, quality hold, and a client requested ship date. Each of the variables can be weighted within a multi-criteria decision making algorithm that maximizes and prioritizes which repair actions should be completed to optimize the overall manufacturing process. In some embodiments, the test plan is automatically re-sequenced to bypass test steps that are impacted by the test failure. For example, in some embodiments, rather than simply stopping the test plan pending a repair action, other test steps of the test plan, that are not impacted by the defect part are executed in parallel while a repair action part is being identified and obtained.
Guidance can be provided through machine learning to dynamically adjust the test plan based on previous experience. In some embodiments, corrective action is performed automatically without the need of interaction by a test operator. For example, a new micro-code update or an automatic re-run of test cases (or sequencing) can be generated using machine learning and experience. In some embodiments, structured and unstructured test result data and defect records are utilized to establish relationships or screens between test steps and root causes, in which the relationships indicate the test steps that are likely to fail in the presence of a particular root cause. The established relationship knowledgebase can then be expanded to take into account multiple root causes. In some embodiments, the relationship knowledge can be modified based on current test results or on new information via one or more cognitive engines for machine learning.
In addition, it is recognized herein that test processes generate an enormous amount of data that can be stored in multiple locations and multiple formats. In one or more embodiments, the cognitive test advisor facility reduces the time required to isolate and repair a failing system or machine. Less time spent de-bugging and analyzing a failure allows, for instance, additional time to be spent repairing herein enhancing test processing by reducing test cycle time and improving product quality.
Yet further, in accordance with one or more aspects, the test advisor facility includes dynamic knowledge transfer processing that dynamically determines using a knowledge transfer score whether a particular recovery task is to be modified by a user to facilitate recovery. This processing restricts a user's input based on the knowledge transfer score having a predefined relationship with one or more knowledge transfer thresholds, allowing the test advisor facility to control certain actions to be taken, without selected input (e.g., selected user input). This aspect, as well as other aspects of the test advisor facility, are further described herein.
In one embodiment, a common application programming interface call can be used across multiple test platforms, if desired. Cognitive test advisor facility 111 processes a user input or input request with reference to one or more databases 112 of historical test data for the manufacturing process. One or more databases 112 of historical test data can be associated with computing resource(s) 110, as shown, or accessible by computing resource(s) 110 across a network. As explained, cognitive test advisor facility 111 returns, in one or more embodiments, suggested test repair actions, which can be ranked repair recommendations that are presented, such as via a graphical user interface (GUI) 120, on a display. One or more of the suggested repair actions are then used by the user, or by the test advisor facility itself directly (depending on the recommendation), to make a repair or an adjustment, such as to the manufacturing process, to address the test failure.
In one or more embodiments, the cognitive test advisor facility is configured to generate test plans based on test failure root causes and/or symptoms. In one or more embodiments, the facility analyzes structured and unstructured data using machine learning processes to determine the best possible test failure repair action recommendation(s), based on user input. Further, in one or more embodiments, solutions to several challenges when querying unstructured data for failure analysis and prioritizing recommendations for repair actions are provided. For example, one challenge is that unstructured data is being queried from databases based on a user-provided input. Thus, in one embodiment, the user input is analyzed and expanded herein, for instance, for non-exact-matched words, spelling errors, technical word data, non-technical word data (e.g., English data), part-related data, etc. Further, another challenge is that a graphical user interface of recommended repair actions for an incident can show multiple recommendations, even though the English meaning of two or more different recommendations in the list may be the same, which can be due to the machine learning processes not understanding the true meaning of the recommendations. Another challenge for the cognitive test advisor facility is to determine relevant and meaningful weights for each queried data record with respect to the user input and/or provided feedback on success or failure of prior repair action recommendations.
In accordance with one or more aspects, a cognitive test advisor facility is provided, which includes, in part, processing to create expanded search queries of manufacturing test data based on user input; to propose recommendations for repair actions that are meaningful, and different from one another; and to provide expanded search input by finding content for identified tag clouds using multiple word/data dictionaries.
In one or more embodiments, the cognitive test advisor is a computer-based facility, tool or system. The computer facility can execute on and/or include one or more processors, for instance, central processing units (CPUs). The processor(s) can include functional components used in the execution of program code, such as functional components to fetch program code from a location, such as cache or main memory, decode program code, and execute program code, access memory for instruction execution, and write results of the executed instructions or code. A processor(s) can also include a register(s) to be used by one or more of the functional components. In one or more embodiments, the computing resource(s) can include memory, input/output, a network interface, and storage, which can include and/or access one or more databases, such as discussed herein. The components of the computing environment can be coupled to each other via one or more buses and/or other connections. Bus connections can be one or more of any of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus, using any of a variety of bus architectures. By way of example, and not limitation, such architectures can include the Industry Standard Architecture (ISA), the Micro-Channel Architecture (MCA), the Enhanced ISA (EISA), the Video Electronic Standard Association (VESA) local bus, and the Peripheral Component Interconnect (PCI). Example computing environments or computer systems which can implement one or more aspects are described further below with reference to
In certain embodiments, some or all of the processes performed by system 200 are performed by one or more specialized computers (e.g., one or more specialized processing units, a specialized computer with a text data analysis component, etc.) for carrying out defined tasks related to machine learning. In some embodiments of the present invention, system 200 and/or components of the system can be employed to solve problems that arise through advancements in technologies such as mentioned herein.
In the example shown in
In some embodiments, test advisor facility 201 constitutes a machine-executable component(s) embodied within a machine(s) (e.g., embodied in one or more computer readable mediums (or media) associated with one or more machines). Such component(s), when executed by the one or more machines, (e.g., computer(s), computing device(s), virtual machine(s), etc.) cause the machine(s) to perform the operations described. In some embodiments, test advisor facility 201 includes a memory 216 that stores computer executable components and instructions. Furthermore, test advisor facility 201 in some embodiments includes one or more processors 218 to facilitate execution of the instructions (e.g., computer executable components and corresponding instructions) by test advisor facility 201. As shown, graphical user interface component 204, query component 206, retrieve and rank component 208, text data analysis component 210, multi-criteria decision making algorithm component 212, knowledge transfer component 213, database(s) 214, memory 216, and/or processor(s) 218 are operatively coupled to one another in one or more embodiments. Again, in other embodiments, test advisor facility 201 includes additional, fewer, and/or other components and/or aspects.
In general, in one or more embodiments, test advisor facility 201 is a cognitive-based tool that is capable of diagnosing a root cause and/or symptom(s) of manufacturing test failures (and/or other test failures) and then recommending one or more corrective course of actions based on structured and unstructured data that is associated with the test failure. In some embodiments, test advisor facility 201 determines one or more test failure repair action(s), e.g., the best possible test failure repair action(s)) for a detected test failure (e.g., manufacturing test failure) by, for example, analyzing data on previously run system tests (e.g., manufacturing systems tests), planned tests, failure information, and/or user feedback data on prior recommended repair actions. Test advisor facility 201 provides, in one embodiment, a heat map that indicates which steps are more likely to cause a failure or be related to a particular symptom. In the context of a test failure, test advisor facility 201, in some embodiments, generates suggested possible repair actions in response to a user interaction with a heat map presented on a graphical user interface. For example, in some embodiments, possible repair actions are provided when a user hovers over the heat map with a mouse cursor or contacts a touch screen of the computer, based on which the suggested action(s) are then provided with supporting keywords. In some embodiments, the graphic user interface provides prioritized recommendations (based on the user input) for repair actions, for instance, repair cycle time for each recommendation. The graphical user interface can also attach test logs to support a recommendation and to assist test technicians.
In order to resolve a test failure, a root cause and/or one or more corrective actions are ascertained. In general, to analyze and resolve test failures, test advisor facility 201, in certain embodiments, identifies certain attributes that are characteristic of a test failure and repair actions that are associated with the test failure via a machine learning service, such as for example, through the use of the IBM Watson® system. IBM and IBM Watson are registered trademarks of International Business Machines Corporation in at least one jurisdiction.
In one embodiment, a graphical user interface is generated that allows a test technician to find test failures during a test process, in which the user is able to provide a set of attributes and a user comment to the system via the graphical user interface (i.e., a user input). In some embodiments, the attributes can include a combination of any of brand, machine type, machine model, commodity, modifier, defect code, operation number, operation step, and user comment fields, as examples. Test advisor facility 201 in certain embodiments is configured to utilize machine learning application programming interfaces and/or services (e.g., IBM Watson application programming interfaces/services, such as a Watson Discovery service and/or a Watson Natural Language Understanding service, that are provided by International Business Machines Corporation), and optionally, one or more multi-criteria decision making algorithm(s) to identify one or more best possible solutions, as discussed herein.
Graphical user interface component 204 of test advisor facility 201 is configured, in one or more embodiments, to generate a heat map pertaining to manufacturing process test failures, in which the heat map is based at least in part on historical test failures and prior repair actions. The heat map can be presented to a user via a graphical user interface that is generated by graphical user interface component 204. After the heat map is generated, test advisor facility 201 could receive a user input 220 via the graphical user interface regarding a test failure of a manufacturing process. In some embodiments, the graphical user interface generated by graphical user interface component 204 has multiple windows and/or views.
As shown in
In one or more embodiments, graphical user interface 400 (e.g., interface component 204 (
In some embodiments, a user may provide feedback via graphical user interface 400 regarding the proposed test repair actions that are presented to the user. In some embodiments, the feedback is provided via the clicking or selecting of a selected symbol, such as an arrow up 434 or an arrow down symbol 436. In some embodiments, the feedback is used to gather information from the user regarding whether a repair action that was displayed in the priority list was successful or unsuccessful. For example, an arrow up 434 can be clicked or selected by the user upon identifying that a repair action of the priority list was successful, and an arrow down 436 can be clicked or selected by the user upon identifying that a repair action of the priority list was unsuccessful. The feedback can be incorporated into machine learning (e.g., deep machine learning supervised or unsupervised algorithms) to improve future recommendation results. In some embodiments, graphical user interface 400 further includes a test log window 438. In some embodiments, test log window 438 is a separate and/or distinct view and/or window from heat map 402. In one or more embodiments, test log window 438 includes one or more interactive icons 440 for selecting a particular database for search (e.g., database(s) 214 (
Returning to
Retrieve and rank component 208 is configured to search for the most relevant test failure records from the queried data and to prioritize the test failure records based on their proximity to user input 220. In some embodiments, retrieve and rank component 208 is configured to utilize a retrieve and rank application programming interface type service to perform the searching, such as for example, the Watson Discovery service and/or the Watson Natural Language Understanding service provided by International Business Machines Corporation. It should be understood that other suitable techniques and/or services offered by International Business Machines Corporation and/or other companies for searching and prioritizing relevant test failure records can be utilized. In some embodiments, retrieve and rank component 208 is configured to search one or more databases 214 for test failure records that are relevant to user input 220 and to prioritize the relevant test failure records via an index score. In one or more embodiments, the index scores are assigned to the relevant test records based at least in part on proximity of the relevant test failure records to user input 220. In some embodiments, retrieve and rank component 208 is configured to detect failure records that have an index score that is higher than a threshold index score, in which the index score is indicative of the similarity and/or proximity of the user input 220 for a particular test failure incident as compared to prior test failures. The proximity can be ascertained using a variety of suitable techniques, such as via clustering techniques, nearest-neighbor techniques, distance vector techniques, and other techniques of similarity or dissimilarity measurement identification. In some embodiments, the search of databases 214 for relevant test failure records is further based at least in part on feedback data that is obtained by querying a feedback storage database (e.g., via query component 206).
Text data analysis component 210 is configured, in one or more embodiments, to analyze text data of records that are identified by retrieve and rank component 208 (e.g., records that have an index higher than a certain threshold value), to find relevant keywords and relevancy rates. The relevant words are those words that have been found in a particular prior test failure. For example, if a certain set of words is often associated with a certain type of test failure, then those words can be relevant to the test failure that is presently being detected by the user. The system associates the relevant keywords with a respective relevancy rate for the given failure occurrence. In some embodiments, text data analysis component 210 utilizes a machine learning engine and/or an application programming interface type service to obtain records from retrieve and rank component 208 that have a retrieve and rank index score that is higher than a certain threshold value. Text data analysis component 210 then analyzes the test data of those obtained records to identify relevant keywords and their respective relevancy rates. For example, in some embodiments, text data analysis component 210 is configured to analyze the unstructured text data of each of the detected failure records to identify relevant keywords and relevancy rates.
Text data analysis component 210 is further configured, in one or more embodiments, to generate a test solution priority list comprising one or more suggested test repair actions for the test failure, in which the list is generated based at least in part on the relevant keywords and the relevancy rates. In some embodiments, text data analysis component 210 is configured to generate the test solution priority list by at least, for each given detected failure record that includes a relevant keyword that has a relevancy rate of the identified relevancy rates that is above a threshold relevancy rate, identifying a test repair action that is stored in the given detected failure record and then counting the frequency of the test repair action across the detected failure records.
In one or more embodiments, based on the proposed solution being a replacement part, test advisor facility 201 is configured to further query an inventory and production database (e.g., via query component 206) to gather data based on a set of attributes that pertain to the viability of a proposed replacement part solution. For example, in some embodiments, a systems applications and products database and/or a materials flow system database is queried. In one or more embodiments, the systems applications and products database and the materials flow system database are within test advisor facility 201 (e.g., database(s) 214). In some embodiments, the set of attributes includes a part availability, a repair cycle time, a repair action difficulty level, a flag on quality hold/stop ship, a requested scheduled ship date, and/or a planned scheduled ship date, as examples. Other examples are also possible.
As noted above, in one embodiment, graphical user interface component 204 is configured to output the test solution priority list to the user by at least displaying, by system 200 via the graphical user interface, one or more suggested test repair actions in a window that is separate and/or distinct from the heat map. In one or more embodiments, outputting the test priority list via the graphical user interface includes displaying, via the graphical user interface, at least one of the respective confidence scores or respective repair cycle times for the one or more test repair actions. In some embodiments, the respective confidence scores or respective repair cycle times are displayed within the window that is separate and/or distinct from the heat map. In some embodiments, the graphical user interface component 204 is configured to output to the user the test solution priority list, feedback inputs, as well as support information and documents.
In one or more embodiments, test advisor facility 201 is configured to execute one or more multi-criteria decision making algorithms (e.g., via multi-criteria decision making algorithm component 212) to narrow down the one or more suggested test repairs of the test solution priority list based at least in part on a plurality of attributes (e.g., via the querying of the inventory and production database in regards to the set of attributes identified above). For example, in some embodiments, multi-criteria decision making algorithm component 212 is configured to obtain the proposed solution(s) of the priority list that are generated by a rank and retrieve type application programming interface (e.g., rank and retrieve component 208), and then to revise the ranking of the proposed solutions in the priority list in view of the identified constraints. In one or more examples, various suitable multi-criteria decision making algorithms can be utilized to prioritize the proposed solutions based on the importance of the plurality of attributes so that a narrowed down priority list can be outputted via, e.g., the graphical user interface to a user. One example of a suitable type of multi-criteria decision making algorithm can be found in Chanchal Saha et al., “A Decision Support System for Real-Time Order Management in a Heterogeneous Production Environment”, Expert Systems with Applications, Oct. 30, 2015, Volume 50. In some embodiments, multi-criteria decision making algorithm component 212 is configured to incorporate manufacturing/inventory constraints (e.g., parts and time availability, requested ship date, and skilled resources attributes), in addition to proposing a revised test solution list that is displayed along with an estimated repair cycle time.
In one or more embodiments, the priority list (e.g., priority list 424 (
In accordance with one or more aspects of the present invention, test advisor facility 201 includes knowledge transfer component 213, which is configured, in one or more embodiments, to determine whether use of a selected input is to be restricted in the determining and/or the performing of tasks (also referred to as activities) to recover from a failure. In one example, the selected input is user input, in which the user is human. In other examples, the user input may be a robot or other user, and/or the selected input may be other than or in addition to user input. As one example, knowledge transfer component 213 is used to determine whether user knowledge is to be used to perform one or more actions to alter execution of one or more tasks, based on a failure of a system (e.g., an information technology system being manufactured). In one example, the altering execution includes performing at least one action that modifies a task or an aspect related to a task, such as, but not limited to, a sequence in which the tasks are executed or selection of an agent (e.g., system, system component, technician, robot, etc.) to perform a task. As examples, the one or more actions include modifying an activity sequence of the one or more tasks to alter the sequence in which the tasks are executed and/or altering a selection of one or more task performing agents to perform the one or more tasks in which one or more other agents are selected to perform the one or more tasks.
In one embodiment, knowledge transfer component 213 is configured to determine a knowledge transfer score and to determine whether the knowledge transfer score meets at least one predefined criterion. For instance, the knowledge transfer score is compared to at least one knowledge transfer threshold to determine whether an action is to be performed. In one example, there is at least one knowledge transfer threshold for each potential action that a user may perform to alter the execution of the one or more tasks. If the knowledge transfer score has a predetermined relationship (e.g., less than or equal) with a selected knowledge transfer threshold, then the user performs an action(s) associated with that selected knowledge transfer. By using the knowledge transfer score, user involvement is restricted in the decision making as the solution evolves. User input is restricted if, for instance, it is determined that a particular solution was satisfactory in the past. This increases the speed of recovery and improves its processing. However, if a solution did not perform as expected or desired, then user involvement may be used. The knowledge transfer score is, in one or more embodiments, a determining factor on the ability of the computer-based system to autonomously perform one or more actions and/or alter execution of one or more actions without any user (e.g., human) intervention.
Additional details of the operation of system 200 (
In some embodiments, structured and unstructured test result data and problem records can be used to establish relationships between test steps and root causes, in which the test steps indicate steps that are likely to fail given the presence of a root cause. In some embodiments, the processes identified above (e.g., process flow 300 of
As noted, in one or more embodiments, the cognitive test advisor facility disclosed herein is configured to generate test plans, and/or suggested test repair actions, based on test value root causes and/or symptoms. In one or more embodiments, the facility analyzes structured and unstructured data using machine learning processes to determine one or more best possible test repair action recommendations, based on the user input. Further, in one or more embodiments, solutions to several challenges when querying unstructured data for failure analysis and prioritizing recommendations for repair actions are provided. For example, in one embodiment, in querying the unstructured data from databases based on a user input, the user input is expanded for non-exact-match words, spelling errors, non-technical word data (e.g., non-English data), part history, etc. In accordance with one or more embodiments, smart queries of manufacturing test data based on user inputs are created, via data analytics, proposing recommendations for repair actions that are meaningful and different from one another, and finding content for the tag clouds using multiple word and/or data dictionaries incorporated as part of, or available to, the test advisor facility.
By way of example,
Continuing with
By way of further explanation,
In one or more aspects, the computer-implemented dynamic technical dictionary structure, or a separate computer-implemented, dynamic technical dictionary data structure, can also be used to derive recommendations for addressing a test failure that are unique, meaningful and potentially different from one another. In particular, in one or more embodiments, the dynamic technical dictionary structure is provided with error codes for the particular manufacturing process tests at issue. Depending on the product family, there may be tens of thousands of error codes for inclusion in the dynamic technical dictionary data structure. Additionally, the dynamic technical dictionary structure is provided with associated reference codes, such as reference code patterns captured in test error messages. In one or more embodiments, the reference codes are each tied or associated with one or more suggested repair actions in a data repository. Further, as noted herein, a relationship strength score indicative of the relationship strength between a particular error code and reference code is provided. These relationships lead to the ability to generate multiple distinct recommendations that are unique. Using this dynamic technical dictionary data structure, the relationship strengths between an error code and a reference code can determine which possible repair action recommendations are to be provided back to the user by the test advisor facility. Note in this regard that a single error code can refer to multiple reference codes, and a single reference code can refer to multiple possible repair actions. Further, note that multiple error codes can refer to the same reference code, depending on the implementation and the repair action required.
As illustrated in
In one or more embodiments, reinforcement learning can use a reinforcement learning score, or hierarchical reinforcement learning score, Zi determined as:
where:
For example, for a test failure, the test advisor facility might provide a repair recommendation (i) as “retry XX00YYY DIMM CD/MM 32 GB MEM”. The user's (k) feedback (ri) for that repair recommendation at various times can be expressed as:
As illustrated in
In accordance with one or more aspects of the present invention, a test advisor facility (e.g., test advisor facility 201; also referred to as a testing system) is configured (e.g., programmed), in one or more embodiments, to determine for a derived recommendation for a system failure, one or more activities or tasks to be performed to execute the recommendation. Further, for each activity, a task performing agent is selected. Moreover, in accordance with one or more aspects of the present invention, the test advisor facility is configured to restrict a selected input (e.g., user knowledge transfer) in select decision making as a failure recovery solution evolves in its definition. One or more of these aspects are described further with respect to
Referring to
Based on the collected data, one or more recommendations for a test failure are derived 1106, and one or more activities are determined that are to be used to execute the one or more recommendations 1108, as described herein in one or more embodiments.
As one example, word2vec and K-means clustering are used to derive recommendations and determine activities (tasks) to execute for the recommendations. For instance, unstructured text (e.g., replace I/O card; emergency power off cycle machine; restart; swap input/output card; reboot the machine; insert new I/O card; restart the operation; run the operation again; power cycle the machine; etc.) is input to a classification technique, such as word2vec and/or K-means clustering, which outputs classifications, including classes (e.g., recommendations) and suggested tasks for those recommendations. Example classifications for the example input provided above include:
Class—Restart machine, with suggested tasks including:
Class—Input/output (I/O) card swap, with suggested tasks including:
Class—retry operation (op), with suggested tasks including:
In another example, machine learning is used to derive recommendations and determine activities (tasks) to execute for the recommendations. For instance, a selected supervised machine learning algorithm of a number of supervised machine learning algorithms receives an input, such as failure scenario data, and provides an output, such as one or more recommended solutions. For instance, an input of failure scenario data (e.g., driver=Driver1; operation OpX; machine type=Type1; hardware under test=power system; system error code=ErrorCode; time insight index (1 to 10): 2; number of tested machine: 10; etc.) is provided to a selected supervised machine learning algorithm and an output of solution classes (e.g., class—restart machine—81%; class—I/O Card swap—10%; class—retry op—9%; etc.) is provided. The machine learning algorithm also provides, in one embodiment, suggested tasks for each recommendation, examples of which include: for restart machine—emergency power off cycle machine, restart, reboot the machine, and power cycle the machine; for I/O card swap—replace I/O card, swap I/O card, and insert new I/O card; and for retry operation (op)—restart the operation, and run the operation again.
Although examples of recommended solutions and tasks to perform those solutions are provided herein, these are only examples. Additional, fewer and/or other recommendations and/or additional, fewer and/or suggested tasks for the recommendations noted herein and/or other recommendations may be provided. The recommendations and tasks provided herein are not meant to be limiting in any way.
Continuing with
Further, in accordance with an aspect of the present invention, a determination is made as to whether selected input (e.g., user knowledge transfer) in select decision making as a failure recovery solution evolves in its definition is to be restricted. For instance, one or more decisions are made as to whether a user (e.g., a human, a robot, etc.) is to perform one or more actions to alter execution of one or more tasks. As used herein, alter execution includes, for instance, altering an execution sequence of one or more tasks and/or altering selection of a task performing agent. However, these are just examples. Altering execution may include other actions in one or more embodiments.
In one example, a determination is made as to whether a user is to modify an activity sequence of the tasks to be performed 1112. As an example, this is determined based on a knowledge transfer score. In one particular example, the knowledge transfer score is determined by comparing the delta between the normalized detractor parameters (e.g., quality commodity index (first pass yield (FPY)), field incident index (field replaceable unit (FRU) log), engineering system test index, etc.) and an F1 score for a repair log repository (e.g., a fabrication test log, a make-to-order test log, etc.). One equation to be used to determine a knowledge transfer score, T, is as follows:
Precision: It is a number of correct positive results divided by the number of positive results predicted by the classifier
Recall: It is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive)
Ki: determines whether a parameter is relevant
Di: Normalized detractor parameters score (0 to 100) [Quality indexes, such as mean time to failure, FRUs (field replacement units) mean time between failures, etc.]
I: Total number of parameters
Xj: Knowledge transfer threshold values, where j=performing agent or list of sequence
wi: Importance of a detractor parameter [i=1 . . . I]
Examples of using K(i), D(i), w(i) values are depicted in
In one example, the F1 score is a harmonic mean between precision and recall. The range of F1 Score is e.g., [0, 100], and it illustrates how precise the classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances). To determine F1, in one example, a search of the historical failures is performed, where the solution steps match the current suggestion by the system. A subset of those historical failures is taken, where they match the failure scenario by at least M, described below, and that subset is called S.
M is used to determine if a failure scenario is new, where new means that this failure scenario is very different than any other seen historically.
Threshold=M; a sample M=0.75=75%
Given the list of all historical failures with n input scenario parameters paired with the list of actions taken to resolve, if at least M of the n input parameters match the failure in question, there is a match and the failure is not new. If it is new, the record is automatically injected (e.g., by test advisor facility 201) into the training data set.
An example calculation of F1 is shown below:
All historical Data (Training)
Failure Scenario in Question
As one example, if T>Xj, then no user input is allowed; and if T<=Xj, then user input is suggested. The value of a threshold, Xj, is based, for instance, on the action to be performed (e.g., type of performing agent or sequence list). As a particular example, Xj is set equal to a selected value, such as 80.00. Other threshold values are also possible.
Returning to 1112, if user input is suggested, then a user is to provide suggested modifications for the activity execution sequence 1114, and processing continues at 1104. Further, in one example, a determination is made as to whether a user is to alter selection of a task performing agent for a particular activity 1116. Again, if user input is suggested, then the user can suggest a change to the selection of task performing agent 1118, and processing continues at 1104. Returning to inquiries 1112, 1116, if it is determined that the user input is not to be accepted (i.e., it is restricted), then an indication is provided that the user input is not to be accepted. In one example, this indication is a refraining by the test advisor facility from performing the one or more actions that would have been suggested by the user if its input was accepted (e.g., modification of task execution sequence, modification of selection of task performing agent). In one or more other examples, the indication may be a log of the restricted input; many examples are possible.
Additionally, if no user input is to be used or if one or more actions are performed based on accepted user input, processing continues with tracking, by the test advisor facility, the progress of each task, verifying completion and waiting for user input, if any, to begin the next task 1120. Based on completing the process, the system records a successful sequence of actions for a failure for future reference, assuming the sequence is considered successful 1122.
As described herein, in one or more aspects, a test advisor facility is provided that leverages restriction of a selected input (e.g., user input) related to execution of tasks to fulfill a recommendation to correct for a failure of a system. In one embodiment, a testing system (e.g., test advisor facility 201) obtains (e.g., determines, is provided, retrieves, generates) an indication of one or more tasks to be performed, based on a failure of a system (e.g., an information technology system being manufactured). The testing system automatically determines whether a user is to perform one or more actions to alter execution of the one or more tasks. The automatically determining is based on an indication of whether a knowledge transfer score of the user meets at least one predefined criterion (e.g., does the knowledge transfer score have a selected relationship (e.g., less than or equal) with respect to at least one selected knowledge threshold—in one example, there is one threshold per potential action to be suggested by the user). Based on the knowledge transfer score meeting the at least one predefined criterion, the one or more actions are performed. As examples, the one or more actions include modifying an activity sequence of the one or more tasks and/or altering a selection of a task performing agent to perform at least one task of the one or more tasks.
However, in one embodiment, based on the knowledge transfer score failing to meet the at least one predefined criterion, an indication is provided, by, e.g., the testing system, that the one or more actions are not to be performed. For instance, the indication includes refraining from performing the one or more actions.
In one example, when it is determined that the user is to perform an action, a graphical user interface, of or coupled to the test advisor facility, allows users to interact with the test advisor facility for decision improvements and to give a check list of actions to fix any test failures. The user can, e.g., edit the order and/or the performing agent of each step of a prescribed list of actions. The user can mark a prescribed list of actions as successful or not. When marked as unsuccessful, this record is updated and added to the training data set for improvement. Within the check list of actions, anything that is found to be solved autonomously by the system, is executed by that system.
In one or more aspects, based on occurrence of a test failure (or failure at another time), the test advisor facility automatically detects the reason of failure and proposes one or more recommendations, e.g., without any user inputs (e.g., absent human inputs). The test advisor facility also proposes, in one example, a list of actions and their respective task performing agent to correct the failure. Prior to executing those tasks, in accordance with one or more aspects, the test advisor facility calculates a knowledge transfer score and compares it to a threshold value. If the knowledge transfer score is, e.g., lower than or equal to the threshold value, then the test advisor facility allows the user to modify/change an activity sequencing and/or task performing agent. By the knowledge transfer process, the test advisor facility learns from the user experience/activity and thereby, improves the knowledge transfer score with the passing of time. Eventually, this process enables the test advisor facility to automatically make decisions upon test failure and execute test repair actions without user intervention. One or more aspects facilitate machine learning from user activities and becoming autonomous over time. In one aspect, the test advisor facility is able to confirm that an action completed successfully through the use of one or more verification tools, such as a floor control system, and/or other systems.
As described herein, cognitive technology and/or machine learning are used in one or more aspects of the present invention. Machine learning is used, for instance, to collect data regarding a failure, solutions, etc. and to evaluate the data to continue learning. Machine learning algorithms generate and train algorithms to create a model utilized to train and/or evaluate recovery solutions. For instance, in an initialization stage, program code (e.g., hardware and/or software) trains these algorithms, based on, for instance, patterns and/or other data.
In identifying various states and/or behaviors indicative of states in the ML training data 1310, the program code can utilize various techniques including, but not limited to, mutual information, which is an example of a technique that can be utilized to identify features (e.g., elements, patterns, attributes, etc.). Further, varying techniques may be used to select features, including but not limited to, diffusion mapping, principal component analysis, recursive feature elimination (an approach to selecting features), and/or a Random Forest, to select the attributes related to various subject states (e.g., training/test data). The program code may utilize a machine learning algorithm 1340 to train the machine learning model 1330 (e.g., the algorithms utilized by the program code), including providing weights for the conclusions, so that the program code can train the predictor functions that comprise the machine learning model 1330. The conclusions may be evaluated by a quality metric 1350. By selecting a diverse set of ML training data 1310, the program code trains the machine learning model 1330 to identify and weight various attributes (e.g., features, patterns, data) that correlate to various states of a subject (e.g., failures, solutions, the environment, etc.).
The model generated by the program code is self-learning as the program code updates the model based on active feedback received from subjects (e.g., the test advisor process, users, and/or the environment). For example, when the program code determines that there is material to be learned that was not previously predicted by the model, the program code utilizes a learning agent to update the model to reflect that information, in order to improve predictions in the future. Additionally, when the program code determines that a prediction is incorrect, either based on receiving user feedback through an interface or based on continuously monitoring the subject, the program code updates the model to reflect the inaccuracy of the prediction for the given period of time. Program code comprising a learning agent cognitively analyzes the data deviating from the modeled expectations and adjusts the model in order to increase the accuracy of the model, moving forward.
In one or more embodiments, program code executing on one or more processors, utilizes an existing cognitive analysis tool or agent to tune the model, based on data obtained from the various data sources. One or more embodiments utilize, for instance, the IBM Watson® system as the cognitive agent; however, other cognitive agents may be used. In one or more embodiments, the program code interfaces with, e.g., IBM Watson application programming interfaces (APIs) to perform a cognitive analysis of obtained data. Again, other cognitive agents and/or other program code interfaces of International Business Machines Corporation™ and/or of other companies may be used. IBM, IBM Watson, International Business Machines Corporation, Developer Cloud (mentioned below) and TrueNorth (mentioned below) are registered trademarks or trademarks of International Business Machines Corporation in at least one jurisdiction.
In one or more embodiments, the program code trains aspects of one or more IBM Watson application program interfaces to learn the material being taught. Utilizing an existing cognitive agent, such as the IBM Watson system, expands the type of subject data that the program code can integrate into the model. For example, data can include documentary, text, visual, and audio data, which the program code can process, based on its utilization of the IBM Watson system. Specifically, in one or more embodiments, certain of the application programming interfaces of the IBM Watson system comprise a cognitive agent (e.g., learning agent) that includes one or more programs, including, but not limited to, natural language classifiers, a retrieve and rank or discovery service (i.e., one or more services available through the IBM Watson® Developer Cloud™ (or others) that can surface the most relevant information from a collection of documents), concepts/visual insights, trade off analytics, document conversion, and/or relationship extraction. In an embodiment, one or more programs analyze the data obtained by the program code across various sources utilizing one or more of a natural language classifier, retrieve and rank application programming interfaces, and trade off analytics application programming interfaces. The IBM Watson application program interfaces can also provide audio related application programing interface services, in the event that the collected data includes audio, which can be utilized by the program code, including but not limited to natural language processing, text to speech capabilities, and/or translation.
In one or more embodiments, the program code utilizes a neural network to analyze the collected, learned and/or generated data to generate the model utilized for training and/or evaluation. Neural networks are a biologically-inspired programming paradigm which enable a computer to learn from observational data, learned data, generated data, and/or other data. This learning is referred to as deep learning, which is a set of techniques for learning in neural networks. Neural networks, including modular neural networks, are capable of pattern (e.g., state) recognition with speed, accuracy, and efficiency, in situations where data sets are multiple and expansive, including across a distributed network, including but not limited to, cloud computing systems. Modern neural networks are non-linear statistical data modeling tools. They are usually used to model complex relationships between inputs and outputs or to identify patterns (e.g., states) in data (i.e., neural networks are non-linear statistical data modeling or decision-making tools). In general, program code utilizing neural networks can model complex relationships between inputs and outputs and identify patterns in data and to generate conversation and/or evaluate such conversation. Because of the speed and efficiency of neural networks, especially when parsing multiple complex data sets, neural networks and deep learning provide solutions to many problems in multiple source processing, which the program code in one or more embodiments accomplishes when obtaining data and generating a model for evaluation and/or training of artificial intelligence-based processes.
One or more embodiments may utilize a neural network to predict future states of a given subject (e.g., predict future performance). Utilizing the neural network, the program code can predict the likelihood of the given state at a subsequent time. The program code obtains (or derives) data related to the subject and/or content being learned from various sources to generate an array of values (possible states) to input into input neurons of the neural network. Responsive to these inputs, the output neurons of the neural network produce an array that includes the predicted states. The program code can automatically transmit notifications related to the predicted states based on the perceived validity.
In one or more embodiments, a neuromorphic processor or trained neuromorphic chip can be incorporated into the computing resources executing the program code. One example of a trained neuromorphic chip that is utilized in an embodiment of the present invention is the IBM® TrueNorth™ chip, produced by International Business Machines Corporation. Other trained neuromorphic chips produced by International Business Machines Corporation and/or other companies may be used.
The IBM TrueNorth chip, also referred to as TrueNorth, is a neuromorphic complementary metal-oxide-semiconductor (CMOS) chip. TrueNorth includes a manycore network on a chip design (e.g., 4096 cores), each one simulating programmable silicon “neurons” (e.g., 256 programs) for a total of just over a million neurons. In turn, each neuron has, e.g., 256 programmable synapses that convey the signals between them. Hence, the total number of programmable synapses is just over, e.g., 268 million (2∧28). Memory, computation, and communication are handled in each of the neurosynaptic cores, so TrueNorth circumvents the von-Neumann-architecture bottlenecks and is very energy-efficient.
One or more aspects of the present invention are inextricably tied to computing and improve processing within a computing environment. By providing and using a test advisor facility, performance of a failure recovery process is expedited and/or improved, improving processing within a computing environment.
Further example embodiments of computing environments to implement one or more aspects of the present invention are described below with reference to
By way of further example,
Computing system 1412 can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computing system. Generally, program modules include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.
As depicted in
In one embodiment, processor 1416 may be based on the z/Architecture® hardware architecture offered by International Business Machines Corporation, or other architectures offered by International Business Machines Corporation or other companies.
Bus 1418 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Computing system 1412 can include a variety of computer system readable media. Such media may be any available media that is accessible by computing system 1412, and it includes both volatile and non-volatile media, removable and non-removable media.
System memory 1423 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1430 and/or cache memory 1432. Computing system 1412 can further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 1434 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media could be provided. In such instances, each can be connected to bus 1418 by one or more data media interfaces. As described below, memory 1423 can include at least one program product having a set (e.g., at least one) of program modules or code that are configured to carry out the functions of embodiments of the invention.
Program/utility 1440, having a set (at least one) of program modules 1442, can be stored in memory 1423 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a networking environment. Program modules 1442 generally carry out the functions and/or methodologies of embodiments of the invention as described herein. Further, a facility, tool, module, logic, system, etc., 1401 can be provided within computing environment 1412 implementing cognitive test advisor facility processing, as disclosed herein.
Computing system 1412 can also communicate with one or more external devices 1414 such as a keyboard, a pointing device, a display 1424, etc.; one or more devices that enable a user to interact with computing system 1412; and/or any devices (e.g., network card, modem, etc.) that enable computing system 1412 to communicate with one or more other computing devices. One particular external device 1414 is a data storage device 1450 that includes, for instance, programs 1452; computer readable program instructions 1454 that may be used to implement one or more aspects of the present invention; and/or data 1456.
Communication can occur via Input/Output (I/O) interfaces 1422. Still yet, computing system 1412 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter 1420. As depicted, network adapter 1420 communicates with the other components of computing system 1412, via bus 1418. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computing system 1412. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.
One or more aspects may relate to cloud computing.
It is to be understood that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.
Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.
Characteristics are as follows:
On-demand self-service: a cloud consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.
Broad network access: capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).
Resource pooling: the provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that the consumer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.
Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.
Service Models are as follows:
Software as a Service (SaaS): the capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based e-mail). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.
Platform as a Service (PaaS): the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.
Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).
Deployment Models are as follows:
Private cloud: the cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on-premises or off-premises.
Community cloud: the cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may exist on-premises or off-premises.
Public cloud: the cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.
Hybrid cloud: the cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).
A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.
Referring now to
Referring now to
Hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.
Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.
In one example, management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.
Workloads layer 90 provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and test advisor processing 96.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Aspects of the present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be accomplished as one step, executed concurrently, substantially concurrently, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
In addition to the above, one or more aspects may be provided, offered, deployed, managed, serviced, etc. by a service provider who offers management of customer environments. For instance, the service provider can create, maintain, support, etc. computer code and/or a computer infrastructure that performs one or more aspects for one or more customers. In return, the service provider may receive payment from the customer under a subscription and/or fee agreement, as examples. Additionally, or alternatively, the service provider may receive payment from the sale of advertising content to one or more third parties.
In one aspect, an application may be deployed for performing one or more embodiments. As one example, the deploying of an application comprises providing computer infrastructure operable to perform one or more embodiments.
As a further aspect, a computing infrastructure may be deployed comprising integrating computer readable code into a computing system, in which the code in combination with the computing system is capable of performing one or more embodiments.
As yet a further aspect, a process for integrating computing infrastructure comprising integrating computer readable code into a computer system may be provided. The computer system comprises a computer readable medium, in which the computer medium comprises one or more embodiments. The code in combination with the computer system is capable of performing one or more embodiments.
Although various embodiments are described above, these are only examples. For example, other machine learning techniques may be used. Further, other types of failures, recoveries, tasks, agents, etc. may be used. Many variations are possible.
Various aspects are described herein. Further, many variations are possible without departing from a spirit of aspects of the present invention. It should be noted that, unless otherwise inconsistent, each aspect or feature described herein, and variants thereof, may be combinable with any other aspect or feature.
Further, other types of computing environments can benefit and be used. As an example, a data processing system suitable for storing and/or executing program code is usable that includes at least two processors coupled directly or indirectly to memory elements through a system bus. The memory elements include, for instance, local memory employed during actual execution of the program code, bulk storage, and cache memory which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
Input/Output or I/O devices (including, but not limited to, keyboards, displays, pointing devices, DASD, tape, CDs, DVDs, thumb drives and other memory media, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the available types of network adapters.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising”, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below, if any, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of one or more embodiments has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain various aspects and the practical application, and to enable others of ordinary skill in the art to understand various embodiments with various modifications as are suited to the particular use contemplated.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7730364 | Chang et al. | Jun 2010 | B2 |
| 7779349 | Spangler | Aug 2010 | B2 |
| 8205173 | Wu et al. | Jun 2012 | B2 |
| 10120912 | Shah-Hosseini | Nov 2018 | B2 |
| 10338993 | Lekivetz et al. | Jul 2019 | B1 |
| 10761974 | Hoover et al. | Sep 2020 | B2 |
| 20080127089 | Peretz | May 2008 | A1 |
| 20170242784 | Heorhiadi et al. | Aug 2017 | A1 |
| 20170262360 | Kochura et al. | Sep 2017 | A1 |
| 20190146901 | Hoover et al. | May 2019 | A1 |
| Entry |
|---|
| Mell, Peter and Tim Grance, “The NIST Definition of Cloud Computing,” National Institute of Standards and Technology, Information Technology Laboratory, Special Publication 800-145, Sep. 2011, pp. 1-7. |
| Sabbagh, Ramin et al., “A Framework Based on K-Means Clustering and Topic Modeling for Analyzing Unstructured Manufacturing Capability Data,” Journal of Computing and Information Science in Engineering, vol. 20, Mar. 2020, pp. 011005-1 thru 011005-13. |
| Madan, Neeraj et al., “Cognitive Test Advisor Facility for Identifying Test Repair Actions,” U.S. Appl. No. 16/744,649, filed Jan. 16, 2020, pp. 1-68. |
| Saha et al., “A Decision Support System for Real-Time Order Management in a Heterogeneous Production Environment”, Expert Systems with Applications, vol. 50, Oct. 2015 (22 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20220188181 A1 | Jun 2022 | US |
