Patent Application
20200293565
Service repairs are often performed to maintain assets such as devices, equipment, machinery, infrastructure, utilities, and the like, which are used in industry, business, government, and residential installations. Service repairs may be performed in response to a service request that has been provided because of a failure or malfunction with an asset. As another example, service repairs may include preventive or scheduled maintenance as a cost-effective practice to ensure that the asset does not fail. Typically, a service request can be entered by a customer while a repair action may be created by a technician after he or she has addressed the service request.
Recently, machine learning has been used to remotely troubleshoot and provide diagnostics of service requests raised by a customer. The machine learning is typically trained using historical services requests and repair actions which can be used to identify patterns between the two. The resulting algorithm can be used to predict a repair or maintenance that is necessary. To convert the service repairs into data that can be used for training purposes, a group of labels (or buckets) must be identified which define different categories of repair actions.
However, a repair action is typically an unstructured text object (free-form text) without any standardized format or language. Instead, terminology that is used within the repair action is chosen subjectively by the engineer/technician that prepared the repair action. As a result, identifying common or standard labels for categorizing the unstructured repair actions can be difficult. Typically, a user (such as a subject matter expert) must manually create a number of labels for the repair actions. This requires the user to read through at least a small set of service repair actions and make a judgment as to which labels should be applied. This can take considerable time. Furthermore, it is infeasible for the user to manually read a large corpus of repair actions (e.g., thousands, etc.). As a result, the labels determined by the user are not very comprehensive and are usually limited to the preferences of the user.
The embodiments herein improve upon the prior art by providing an automated label determination system which can identify a finite set of labels from among a large corpus of service repair actions. Furthermore, based on the identified labels, the system can further categorize each of the service repair actions in a corpus of historical repair actions as being associated with at least one label from among the set of labels and output the results via a user interface. The labels can be used in a subsequent training process for building a supervised learning algorithm for predicting repair actions to be taken.
The service repair actions include descriptions composed of free-form text. Often, a technician or engineer will document the problem, the action taken, and the result, and store this information as a repair action document or text object. The system described herein can receive many historical service repair actions and generate a dictionary of words which include unigram terms extracted from the free-form text of the repair actions. All unique words may initially be extracted and then the list of words may be cleaned or trimmed down by removing various stop words (conjunctions, articles, prepositions, etc.) and generic words that do not discretize between different types of repair actions. The system may create a term matrix based on the remaining words in the dictionary of words.
For example, the term matrix may include a list of words on one axis and the repair actions on another axis, mapped to each other to create an array of cells. The cells may include an identifier that indicates a frequency of use of each word with respect to each repair action. According to various embodiments, the system may execute a non-negative matrix factorization (NMF) operation which can decompose the term matrix into two smaller matrices including a topic-to-keyword matrix and a topic-to-action matrix. The topic-to-keyword matrix may identify a plurality of topics (i.e., a plurality of labels). Furthermore, the topic-to-keyword matrix may indicate which words in the dictionary (keywords) are associated with each respective topic. Meanwhile, the topic-to-action matrix may map the plurality of topics to the plurality of repair actions. The cells of the topic-to-action matrix may indicate a likelihood of a repair action being associated with a corresponding topic. Based on the likelihoods in the topic-to-action matrix, the system can estimate which label (i.e., topic) a repair action is most closely associated with (belongs to).
The learning system may also provide a user interface which allows a user to enter feedback input regarding the labels and the keywords associated with the labels. The system described herein may convert the user feedback into modifications to the topic-to-keyword matrix and the topic-to-action matrix. For example, the system can convert the feedback into instructions which can mathematically modify the content of one or more elements of the topic-to-keyword matrix and/or the topic-to-action matrix. Examples of feedback include addition, deletion, renaming, splitting, and merging of topics/labels. As another example, feedback may include modifying keywords associated with the labels (e.g., modifying a weight given to keywords, etc.) The feedback collection and the matrix modification may be performed by the same software or it may be performed by different pieces of software. For example, a first piece of software may implement the user interface, while a second piece of software receives the feedback input from the user interface, and mathematically modifies the matrices.
In an aspect of an example embodiment, a computing system may include a storage configured to store a plurality of repair action entries which each comprise unstructured free-form text associated with actions performed, and a processor configured to generate a term matrix that comprises a list of words extracted from the plurality of repair action entries and which are mapped to the plurality of repair action entries based on frequency of use within free-form text thereof, determine a plurality of labels for categorizing the actions performed via execution of a non-negative matrix factorization based on the generated term matrix, and output information about the determined plurality of labels for display via a user interface.
In an aspect of another example embodiment, a method may include storing a plurality of repair action entries which each comprise unstructured free-form text associated with actions performed, generating a term matrix that comprises a list of words extracted from the plurality of repair action entries and which are mapped to the plurality of repair action entries based on frequency of use within free-form text thereof, determining a plurality of labels for categorizing the actions performed via execution of a non-negative matrix factorization based on the generated term matrix, and outputting information about the determined plurality of labels for display via a user interface.
Other features and aspects may be apparent from the following detailed description taken in conjunction with the drawings and the claims.
Features and advantages of the example embodiments, and the manner in which the same are accomplished, will become more readily apparent with reference to the following detailed description taken in conjunction with the accompanying drawings.
Throughout the drawings and the detailed description, unless otherwise described, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The relative size and depiction of these elements may be exaggerated or adjusted for clarity, illustration, and/or convenience.
In the following description, specific details are set forth in order to provide a thorough understanding of the various example embodiments. It should be appreciated that various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosure. Moreover, in the following description, numerous details are set forth for the purpose of explanation. However, one of ordinary skill in the art should understand that embodiments may be practiced without the use of these specific details. In other instances, well-known structures and processes are not shown or described in order not to obscure the description with unnecessary detail. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
The example embodiments are directed to a learning system that uses non-negative matrix factorization to learn the approximate number of topics (labels) of corrective repair action seen by a services team, or the like. The learning system can identify similar types of corrective repair actions, cluster them and assign a common label thereto. Furthermore, the learning system provides a mechanism in which feedback provided by a human (e.g., subject matter expert) can be used to mathematically refine an element of the non-negative matrix factorization. The learning system can create a labeled training set of repair actions which can be used for training machine learning models for automated troubleshooting of repair orders in a fast and accurate manner.
Supervised machine learning is often performed to predict what kind of service needs to be performed (e.g., condition-based maintenance, and the like). In order for a machine learning algorithm to be created there must be a labeled set of training data. The machine learning algorithm uses the labeled training data to identify patterns (e.g., what request triggered what action, etc.). The actions are not infinite and may be limited to a smaller set of actions such as 20 actions, 50 actions, 100 actions, or the like.
Within the historical repair action data, the description of the action taken is typically in free-form unstructured text. Furthermore, engineers/technicians may describe the work they performed for handling a service request in different ways. The same action may have a very different description. Therefore, allowing the machine learning to learn can be difficult. The goal is to find a way to create the buckets (labels) for each type of action that was performed and to commonly label actions in the same bucket even when their descriptions are not exactly alike. However, the actions are written in free text which makes it difficult to convert the free-form text into standardized buckets.
To address this problem in an automated manner, the learning system described herein generates a term matrix from a dictionary of words which are acquired from historical repair actions. The term matrix may include a listing of the words on one axis, and the list of repair actions on the other axis. The cells of the term matrix may identify a frequency of use of the words within each repair action. The system may then perform a non-negative matrix factorization to decompose the term matrix into a topic-to-keyword matrix and a topic-to-action matrix. The topic-to-keyword matrix may include keywords mapped to each topic and may be determined based on commonly used words in a cluster of similar repair actions clustered together. Meanwhile, the topic-to-action matrix may include likelihood indicators that a repair action is associated with each respective topic which can be used to estimate a label (topic) from among the plurality of topics for each of the repair actions.
In the example of
The learning system 120 may collect a bunch of historical service action repair records and create a matrix format such as shown in the example of the term matrix 210 shown in
Non-negative matrix factorization (NMF) based labeling may be performed based on the created term matrix. For example, the learning system 120 may define a metric reconK(k) and choose that value of k which minimizes it (examples are shown in
The learning system 120 may also incorporate feedback from a subject matter expert which is received via the user interface 130. Examples of the types of feedback include, but are not limited to, addition of labels (topics), deletion of any unimportant labels, renaming of labels, modifying keywords associated with the labels, merging two or more similar labels together into a single label, and splitting a label that is too generic into more than one label. Each of these options may be output to the user interface 130. In response to receiving any of these feedbacks, the learning system 120 may automatically modify the topic-to-keyword matrix and/or the topic-to-action matrix by modifying the content included in rows and/or columns of the topic-to-keyword matrix and/or the topic-to-action matrix.
The learning system 120 can abstract out key repair issues from the repair action corpus with free-text descriptions of repair actions performed and create a labeled training set with a finite number of labels, which is essential for creating a robust multi-class supervised learning model. The system may use limited human feedback which the system can use to mathematically modify the topic-to-keyword matrix and/or the topic-to-action matrix. Some of the advantages of the learning system include a novel metric to be optimized to determine the number of topics in the corpus, making a first level of labeling scalable, and efficient in terms of speed and manhours utilized. Furthermore, because the clustering of ‘issues’ is done after analyzing the entire corpus, it is more comprehensive unlike a complete manual effort which will iteratively look at few sample cases to come up with mapping rules. Furthermore, the system provides a formalized feedback mechanism to incorporate expert feedback thereby making the process more streamlined.
1. Due to the inherent sparsity constraints in NMF, it is observed (empirically) to work better for short, noisy text. In our case, via an ad-hoc inspection, topics obtained by NMF were qualitatively better than those obtained by LDA.
2. It is easier to incorporate feedback in the NMF based topic modeling approach.
3. LDA is highly sensitive to hyperparameter tuning compared to NMF.
Non-negative factorization is a dimensional reduction technique in which the low rank factor matrices are constrained to be non-negative. Suppose non-negative matrix. The goal of non-negative matrix factorization, with a desired lower dimension of k, is to determine two matrices W∈
n×k and H∈
k×m such that A≈WH. In the example of
where ∥·∥F is defined as the Frobenius norm of the matrix. The above optimization problem (Equation 1) is solved by solving the following two sub-problems alternately until convergence is achieved:
This method is referred to as alternative non-negative least squares (NNLS).
A first axis (y axis) of the term matrix 210 is the dictionary of words after conjunctions, generic terms, and redundancies have been removed. In total, the dictionary includes n words or more generically, n-grams. A second axis (x axis) of the term matrix 210 is the corpus of repair action records. In total, m repair action records are included. Individual cells 211 are created based on the rows of words overlapped with the columns of repair action. Each cell may provide an indicator which indicates how many times a word is included in a repair action. In the examples herein, colors or shading are used to indicate a value within the cells, however, embodiments are not limited thereto. Other examples include numerical values, etc.
The term matrix 210 may be generated from a large data file that includes textual objects or documents of description included within a corpus of repair action records. Examples of the type of data included in the repair actions include, but are not limited to:
1) Service Dispatch Type—nature of the service call (planned, unscheduled, etc.)
2) Service Request Number—unique identifier of each service request.
3) Service Request Activity Number—number of activities taken by a service engineer in an effort to solve the problem, within each Service Request.
4) Problem Found—The problem to be addressed by the service request.
5) Action Taken—The action(s) taken to solve the problem.
6) Verification—Test results to verify the solution taken.
7) Action Code—debrief item to be filled by a service engineer that requires the engineer to choose from a list of predefined action codes.
The learning system herein can perform a plurality of steps in order to convert the initial data file including the corpus of repair actions into the term matrix 210. For example, the learning system may perform one or more of the following:
1) Removal of SR with multiple activities: Many a times, for SR's with multiple activities, there is often discrepancy in the text written for different activities which can lead to confusion. For ensuring clean data in the topic modeling, we use those repair actions which have only one activity excluding ‘No Problem Found’. Since we are interested only in generating the set of labels and the associated labeling function for each label, this filter does not hamper the learning of the labeling process (as long as we have sufficient examples).
2) Removal of duplicates: Using regular expressions, the system may search for variants of the term “duplicates” in the verification text and remove those repair actions.
3) Removal of part replacements: In several repair actions, part replacement may be required. The parts replaced are stored in a separate database. Such repair actions are labeled by the corresponding part replaced and hence are not included in the subsequent process.
4) Removal of No Problem Found: For repair actions where the Action Code corresponds to ‘No Problem Found’ they are removed.
5) Dictionary Creation and Stopword removal: A stopword list is created and includes common conjunctions, articles, preposition as well as domain related unimportant words that do not discern between different topics of action. After removing these words, a dictionary of remaining unigram terms is created after applying standard methods like stemming and lemmatization.
6) Generation of the term matrix 210 which includes a matrix where the columns are labeled with repair actions (m total) and the rows are labeled with the words from the created dictionary (n total) to create a n×m matrix using a TF-IDF method.
7) Removing repair actions with automated messages: Many times, the Action Taken column contains automated messages. Issues related to these messages are already known and hence need to be removed during pre-processing. In absence of any identifier for such messages, the system may use a heuristic method to identify and eliminate them.
is defined as the condition number and σmax, σ1 are the maximal and minimal singular values of A. The reconstruction error ∥A−WH∥F2 is a natural method to choose the best k. However, this has a natural bias to a higher k, and therefore the new metric is used.
Referring to
In the example of
In the example of
The learning system may also provide a user interface (such as user interface 400) shown in
Referring to
The following Table 1 provides a list of six example feedback inputs that can be entered using the user interface 400, and the resulting mathematical operation performed by the learning system described herein.
In 520, the method may include generating a term matrix that comprises a list of words extracted from the plurality of repair action entries and which are mapped to the plurality of repair action entries based on frequency of use within free-form text thereof. The term matrix may include the term matrix 210 shown in the example of
In 530, the method may include determining a plurality of labels for categorizing the actions performed via execution of a non-negative matrix factorization based on the generated term matrix, and in 540, the method may include outputting information about the determined plurality of labels for display via a user interface. In some embodiments, the determining the plurality of labels may include identifying a finite number of repair themes based on a correlation of keywords included in the generated term matrix, and assigning each repair theme as a respective label. For example, the determining may include identifying similar repair action entries based on words in the list of words, clustering the similar types of repair action entries into a cluster, and assigning a label to the clustered repair action entries based on the non-negative matrix factorization.
In some embodiments, the determining may include decomposing the term matrix into a topic-to-keyword matrix that comprises mappings between the list of words and the plurality of labels based on the generated term matrix. In some embodiments, the method may further include decomposing the term matrix into a topic-to-action matrix in which likelihoods that a repair action entry is associated with each of the plurality of labels are identified. In some embodiments, the outputting may further include outputting a respective label for each of the plurality of repair action entries for display via the user interface based on the generated topic-to-action matrix.
In some embodiments, the method may further include receiving user feedback input via a user interface, and modifying the generated topic-to-keyword matrix based on the received user feedback. The user interface may be the same software which is being used to perform the learning. As another example, the user interface may be a separate piece of software from the learning system. The user feedback may include a command that includes one or more of deleting, adding, renaming, merging, and splitting a label. As another example, the command may modify a weight that is applied to keywords. The modifying may include generating and applying instructions which mathematically modify one or more of the topic-to-keyword matrix and the topic-to-action matrix.
The network interface 610 may transmit and receive data over a network such as the Internet, a private network, a public network, and the like. The network interface 610 may be a wireless interface, a wired interface, or a combination thereof. The processor 620 may include one or more processing devices each including one or more processing cores. In some examples, the processor 620 is a multicore processor or a plurality of multicore processors. The input/output 630 may be a hardware device that includes one or more of a port, an interface, a cable, etc., that can receive data input and output data to (e.g., to an embedded display of the device 600, an externally connected display, an adjacent computing device, a cloud platform, a printer, an input unit, and the like. The storage device 640 is not limited to any particular storage device and may include any known memory device such as RAM, ROM, hard disk, and the like.
According to various embodiments, the storage 640 may store a data file, document, or the like, which includes data of a plurality of repair action entries which each comprise unstructured free-form text associated with actions performed. The processor 620 may generate a term matrix that comprises a list of words extracted from the plurality of repair action entries and which are mapped to the plurality of repair action entries based on frequency of use within free-form text thereof. The processor 620 may determine a plurality of labels for categorizing the actions performed via execution of a non-negative matrix factorization based on the generated term matrix. The processor 620 may output information about the determined plurality of labels for display via a user interface. The repair action entries may include unstructured textual descriptions of repair services performed.
According to various embodiments, the processor 620 is configured to determine a finite number of repair themes based on a correlation of keywords included in the generated term matrix, and assign each repair theme as a respective label. For example, the processor 620 may identify similar repair action entries based on words in the list of words, cluster the similar types of repair action entries into a cluster, and assign a label to the clustered repair action entries based on the non-negative matrix factorization.
For example, the processor 620 may decompose the term matrix into a topic-to-keyword matrix that comprises mappings between the list of words and the plurality of labels based on the generated term matrix. In addition, the processor 620 may decompose the term matrix into a topic-to-action matrix in which likelihoods that a repair action entry is associated with each of the plurality of labels are identified. In some embodiments, the processor 620 may output a respective label for each of the plurality of repair action entries for display via the user interface based on the generated topic-to-action matrix. In some embodiments, the processor 620 may output a user interface which receives user feedback input via the user interface, and modify the generated topic-to-keyword matrix based on the received user feedback.
As will be appreciated based on the foregoing specification, the above-described examples of the disclosure may be implemented using computer programming or engineering techniques including computer software, firmware, hardware or any combination or subset thereof. Any such resulting program, having computer-readable code, may be embodied or provided within one or more non-transitory computer-readable media, thereby making a computer program product, i.e., an article of manufacture, according to the discussed examples of the disclosure. For example, the non-transitory computer-readable media may be, but is not limited to, a fixed drive, diskette, optical disk, magnetic tape, flash memory, semiconductor memory such as read-only memory (ROM), and/or any transmitting/receiving medium such as the Internet, cloud storage, the internet of things, or other communication network or link. The article of manufacture containing the computer code may be made and/or used by executing the code directly from one medium, by copying the code from one medium to another medium, or by transmitting the code over a network.
The computer programs (also referred to as programs, software, software applications, “apps”, or code) may include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, cloud storage, internet of things, and/or device (e.g., magnetic discs, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The “machine-readable medium” and “computer-readable medium,” however, do not include transitory signals. The term “machine-readable signal” refers to any signal that may be used to provide machine instructions and/or any other kind of data to a programmable processor.
The above descriptions and illustrations of processes herein should not be considered to imply a fixed order for performing the process steps. Rather, the process steps may be performed in any order that is practicable, including simultaneous performance of at least some steps. Although the disclosure has been described in connection with specific examples, it should be understood that various changes, substitutions, and alterations apparent to those skilled in the art can be made to the disclosed embodiments without departing from the spirit and scope of the disclosure as set forth in the appended claims.
