Patent Application
20160217126
Textual classification is used in many contexts to ascribe one or more characteristics or categories to a set of text. The set of text may simply be a word, a paragraph, or an entire document or set of documents. Automatic textual classification is highly useful in that important information can be determined automatically about the text without requiring a user to read through the text first.
Automatic textual classification, in some contexts, may employ neural networks and/or a Naïve Bayes Classifier. Regardless, typical methods generally require significant computational overhead. In instances where vast text is generated, traditional methods of text classification may be too slow and/or beyond the reasonable capacity of the device on which the classification is performed.
The discussion above is merely provided for general background information and is not intended to be used as an aid in determining the scope of the claimed subject matter.
A system for classifying text is provided. The system includes a data store containing a plurality of previously observed word sequences and a processor coupled to the data store. The processor is configured to receive a first word sequence and generate bi-directional similarity metrics based on the first word sequence and each of the previously observed word sequences. The processor is also configured to assign a classification to the first word sequence based on at least one of the bi-directional similarity metrics.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the background.
Embodiments described herein provide a highly efficient system for classifying text. Current approaches to classifying text involve either vectorizing a set of words and establishing a numerical distance between the vectors or finding metrics such as edit-distance between the set of words. The former suffers from information loss due to normalization while the latter is a non-numeric unidirectional operation that does not take into consideration the information present in both sets of words. Embodiments described below provide a bi-directional similarity metric between two sets of text or word sequences based on information in both word sequences.
In order to proactively identify problems before they affect users 101, a periodic analysis of all of the logs from system 100 may be performed. While such log analysis is useful in identifying potential problems before they affect users, the log analysis operation is very labor intensive. In order to facilitate the analysis of system 100 before users are affected, a listing of all exceptions and their associated exception data can be received from system 100 by exception processor 112, as indicated by arrow 114. When an error occurs, either the system or the currently executing application reports the error by throwing an exception containing information about the error. Once the exception is thrown, it is handled by the application or by a default exception handler. An exception generally includes significant information relative to the error. For example, the exception may include a text message to inform the user of the nature of the error and suggest action to resolve the problem; stack trace information; a help link; a source of the exception, as well as any additional information that may be relevant to the potential cause of the exception. Thus, exceptions can be rich collections of textual information that provide significant insight into system operation during an error.
Exception processor 112 may be a component of system 100 or may be separate therefrom. Exception processor 112 collects a list of all the exceptions that are possible and builds a hyperspace of exceptions, illustrated as points 114, 116, 118 in hyperspace 120. Then, periodically, such as daily, or in response to an event, such as the generation of a new exception, exception processor 112 will compare one or more unclassified exceptions with the known list of exceptions in order to classify the one or more unclassified exceptions. As can be appreciated, if a new exception is very similar to a previously seen exception, the new exception may be classified as related to the previously seen exception. Such similarity may allow a system engineer to assess whether remedial action appropriate for the previously seen exception may also be appropriate or at least similar to appropriate remedial action for the new exception. Conversely, if the new exception is not similar to any previously seen exceptions, then exception processor 112 can automatically escalate the exception to appropriate personnel since it may reflect an entirely new type of problem.
While embodiments described herein will be described in the context of analyzing exceptions, it should be understood that exceptions are simply one example of textual information that is amenable to the classification system and techniques described herein. Embodiments are applicable classifying any textual information and are certainly not limited to exceptions.
UI component 152, in some examples, is able to generate or otherwise facilitate interactions with one or more users in order to allow users to interact with system 100. UI component 152 may generate one or more dialogs or user interface displays to one or more users through any suitable mechanism, such as a local display of a computing device or via a web page using a suitable data, such as HTML data.
System 100 includes or is coupled to network component 154, which allows system 100 to communicate with other devices through a suitable communication network, such as a local area network (LAN), a wide area network (WAN), such as the internet, or a combination thereof. In some examples, network component 154 may include a wired physical layer facilitating communication in accordance with the known Ethernet protocol. However, in other examples, network component 154 may include a wireless communication module(s) in addition to, or instead of, a wired physical layer. Regardless, network component 154 allows system 100 to communicate with one or more user devices 102 through network 160.
System 100 also includes or is coupled to data store 156, which may include a database or other suitable structure for storing a number of textual collections. Some of the textual collections stored within data store 156 may be training data that have already been analyzed and/or characterized. Additionally, data store 156 may include a number of textual collections (such as a log of exception data) for which classification is required.
Classifier 112 obtains a collection of text, such as an exception or other suitable grouping of text, and classifies the text by determining a bi-directional similarity metric for the collection of text as compared to one or more previously classified collections of text. Classifier 112 may also include or receive a similarity threshold such that if the similarity of the collection of text and one or more previously-classified collections of text is above the similarity threshold, then the collection of text may be assigned a classification. In one example, the collection of text includes exception information. However, embodiments are applicable to a variety of collections of text ranging from a couple of words or sentences to entire documents or collections of documents.
At block 212, a bi-directional similarity metric is calculated between the first and second word specimens. The similarity metric is bi-directional in the sense that if the text of one specimen is fully encompassed in the second specimen, but the second specimen contains additional text not found in the first specimen, then the metric will result in less than a perfect match. Only if both specimens match each other identically, will the bi-directional similarity metric return a perfect result. In one example, the bi-directional similarity metric provides a probability that the word specimens or sequences are the same. More formally, P(Word Sequence 1 and 2 are same)=P(Word Sequence 1 is similar to Word Sequence 2)*P(Word Sequence 2 is similar to Word Sequence 1). The probability that Word Sequence 2 is similar to Word Sequence 1 is given by the total number of words in Word Sequence 2 that exist in Word Sequence 1 divided by the total number of words in Word Sequence 2. Similarly, the probability that Word Sequence 1 is similar to Word Sequence 2 is given by the total number of words in Word Sequence 1 that exist is Word Sequence 2 divided by the total number of words in Word Sequence 1. As set forth above, these two probabilities are combined, such as by multiplying them together, in order to provide the bi-directional similarity metric. However, embodiments also include applying weighting factors such that one direction is favored more than the other.
At block 214, the bi-directional similarity metric determined at block 212 is compared with a pre-defined threshold in order to determine whether to apply a classification to the first word sequence or specimen. If the bi-directional similarity metric is above the pre-defined threshold, then the classification is applied to the first word sequence, as indicated at block 216. Conversely, if the bi-directional similarity metric is not above the pre-defined threshold, then the first word sequence is not classified, and control passes to block 218, where control may return to block 202 via dashed line 220 to compare the first word sequence to another word sequence. In this way, the first word sequence will generally be classified based on its nearest neighbor in the collection.
The pre-defined threshold to cut-off classification can be learned or otherwise determined using training data and one or more binary classifiers can be trained to match a document/exception with various stack traces/word sequences. While training the classifier, the bias variance tradeoff is incorporated through the threshold value. As can be appreciated, selection of the threshold value will determine the cluster density. For example, lower thresholds will result in broader clusters while higher thresholds will result in tighter clusters.
Embodiments described herein are able to quickly utilize previously seen examples of text in order to classify new sets of text. However, embodiments can also be used to dynamically generate clusters of text. For example, a dynamic cluster can be started with a null set and will involve either the addition of a word sequence or incrementing an existing sequence's counter value depending on the threshold probability of a match between the two word sequences. As described above, a lower threshold will result is broader clusters while a higher threshold will result in tighter clusters.
The present discussion has mentioned processors and servers. In one embodiment, the processors and servers include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.
Also, embodiments described herein may employ a variety of user interface displays. Such user interface displays may have different forms and a wide variety of different user actuatable input mechanisms disposed thereon. For instance, the user actuatable input mechanisms can be text boxes, check boxes, icons, links, drop-down menus, search boxes, etc. They can also be actuated in a wide variety of different ways. For instance, they can be actuated using a point and click device (such as a track ball or mouse). They can be actuated using hardware buttons, switches, a joystick or keyboard, thumb switches or thumb pads, etc. They can also be actuated using a virtual keyboard or other virtual actuators. In addition, where the screen on which they are displayed is a touch sensitive screen, they can be actuated using touch gestures. Also, where the device that displays them has speech recognition components, they can be actuated using speech commands.
A number of data stores have also been discussed. It will be noted such data stores can each be broken into multiple data stores. All can be local to the systems accessing them, all can be remote, or some can be local while others are remote. All of these configurations are contemplated herein.
Also, the figures show a number of blocks with functionality ascribed to each block. It will be noted that fewer blocks can be used so the functionality is performed by fewer components. Also, more blocks can be used with the functionality distributed among more components.
The description is intended to include both public cloud computing and private cloud computing. Cloud computing (both public and private) provides substantially seamless pooling of resources, as well as a reduced need to manage and configure underlying hardware infrastructure.
A public cloud is managed by a vendor and typically supports multiple consumers using the same infrastructure. Also, a public cloud, as opposed to a private cloud, can free up the end users from managing the hardware. A private cloud may be managed by the organization itself and the infrastructure is typically not shared with other organizations. The organization still maintains the hardware to some extent, such as installations and repairs, etc.
In the embodiment shown in
It will also be noted that computing system 100, or portions of it, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, et cetera.
Computer 810 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 810 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media is different from, and does not include, a modulated data signal or carrier wave. It includes hardware storage media including both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 810. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 830 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 831 and random access memory (RAM) 832. A basic input/output system 833 (BIOS), containing the basic routines that help to transfer information between elements within computer 810, such as during start-up, is typically stored in ROM 831. RAM 832 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 820. By way of example, and not limitation,
The computer 810 may also include other removable/non-removable volatile/nonvolatile computer storage media. By way of example only,
Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
The drives and their associated computer storage media discussed above and illustrated in
A user may enter commands and information into the computer 810 through input devices such as a keyboard 862, a microphone 863, and a pointing device 861, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 820 through a user input interface 860 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A visual display 891 or other type of display device is also connected to the system bus 821 via an interface, such as a video interface 890. In addition to the monitor, computers may also include other peripheral output devices such as speakers 897 and printer 896, which may be connected through an output peripheral interface 895.
The computer 810 is operated in a networked environment using logical connections to one or more remote computers, such as a remote computer 880. The remote computer 880 may be a personal computer, a hand-held device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 810. The logical connections depicted in
When used in a LAN networking environment, the computer 810 is connected to the LAN 871 through a network interface or adapter 870. When used in a WAN networking environment, the computer 810 typically includes a modem 872 or other means for establishing communications over the WAN 873, such as the Internet. The modem 872, which may be internal or external, may be connected to the system bus 821 via the user input interface 860, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 810, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
Embodiments described herein allow a computer system to perform classification of text very quickly with relatively little computational overhead. The classification is based on a bi-directional similarity metric that employs information from both word sequences in an efficient operation. Thus, the computer is able to perform the textual classification faster than would otherwise be possible. Additionally, in embodiments where classification is performed on exception information, or other computer-generated textual information, more significant exceptions can be automatically classified and surfaced automatically for remedial action. This reduces the time required by technicians or operators to read through all of the exception information first or even at all.
It should also be noted that the different embodiments described herein can be combined in different ways. That is, parts of one or more embodiments can be combined with parts of one or more other embodiments. All of this is contemplated herein. Various examples are set forth below.
Example 1 is system for classifying text, the system includes a data store containing a plurality of previously observed word sequences and a processor coupled to the data store. The processor is configured to receive a first word sequence and generate bi-directional similarity metrics based on the first word sequence and each of the previously observed word sequences. The processor is also configured to assign a classification to the first word sequence based on at least one of the bi-directional similarity metrics.
Example 2 is a system for classifying text of any or all of the previous examples, wherein a respective bi-directional similarity metric is based on a number of words of the first word sequence that are present in a respective one of the plurality of previously observed word sequences as well as the number of words in the respective one of the plurality of previously observed word sequences that are present in the first word sequence.
Example 3 is a system for classifying text of any or all of the previous examples, wherein each respective bi-directional similarity metric is based a probability of similarity that the first word sequence is similar to a respective one of the plurality of previously observed word sequences in combination with a probability of similarity that the respective one of the plurality of previously observed word sequences is similar to the first word sequence.
Example 4 is a system for classifying text of any or all of the previous examples, wherein the probability of similarity that the first word sequence is similar to a respective one of the plurality of previously observed word sequences is based on a ratio of a total number of words of the first word sequence that are present in the respective one of the plurality of previously observed word sequences to the total number of words in the respective one of the plurality of previously observed word sequences.
Example 5 is a system for classifying text of any or all of the previous examples, wherein the probability of similarity that a respective one of the plurality of previously observed word sequences is similar to the first word sequence is based on a ratio of a total number of words of the respective one of the plurality of previously observed word sequences that are present in the first word sequence to the total number of words in the first word sequence.
Example 6 is a system for classifying text of any or all of the previous examples, wherein the bi-directional similarity metric is the product of the probability of similarity that the first word sequence is similar to a respective one of the plurality of previously observed word sequences and the probability of similarity that the respective one of the plurality of previously observed word sequences is similar to the first word sequence.
Example 7 is a system for classifying text of any or all of the previous examples, wherein the product includes equal weights for each of the probabilities.
Example 8 is a system for classifying text of any or all of the previous examples, wherein the probability of similarity that a respective one of the plurality of previously observed word sequences is similar to the first word sequence is based on a ratio of a total number of words of the respective one of the plurality of previously observed word sequences that are present in the first word sequence to the total number of words in the first word sequence.
Example 9 is a system for classifying text of any or all of the previous examples, wherein the processor is configured to perform pre-processing of the first word sequence before determining the bi-directional similarity metrics.
Example 10 is a system for classifying text of any or all of the previous examples, wherein the pre-processing includes removing stop words.
Example 11 is a system for classifying text of any or all of the previous examples, wherein pre-processing includes maintaining multiple occurrences of the same word.
Example 12 is a system for classifying text of any or all of the previous examples, wherein pre-processing includes alphabetizing the first word sequence.
Example 13 is a system for classifying text of any or all of the previous examples, wherein the plurality of previously observed word sequences are pre-processed.
Example 14 is a system for classifying text of any or all of the previous examples, wherein the first word sequence is computer-generated.
Example 15 is a system for classifying text of any or all of the previous examples, wherein the computer-generated first word sequence includes exception information.
Example 16 is a system for classifying text of any or all of the previous examples, wherein the processor is configured to selectively apply the classification if at least one of the bi-directional similarity metrics exceeds a pre-defined threshold.
Example 17 is a computer-implemented method for classifying computer-generated text. The method includes pre-processing the computer-generated text and at least one previously observed exception. A first probability of similarity of the computer-generated text to the at least one previously observed exception is determined. A second probability of similarity of the at least one previously observed exception to the computer-generated text is determined. A bi-directional similarity metric is generated based on the first and second probabilities. The computer-generated text is selectively classified if the bi-directional similarity metric exceeds a pre-defined threshold.
Example 18 is a computer-implemented method of any or all of the previous examples wherein the computer-generated text is exception information.
Example 19 is a computer-implemented method of comparing a first set of text to a second set of text. The method includes determining a first probability of similarity of the first set of text to the second set of text and determining a second probability of similarity of the second set of text to the first set of text. A bi-directional similarity metric is generated based on the first and second probabilities. The first set of text is classified based on the bi-directional similarity metric and the second set of text.
Example 20 is a computer-implemented method of any or all of the previous examples wherein the first probability is based on a total number of words in the first set of text that are present in the second set of text divided by the total number of words in the first set of text; and the second probability is based on a total number of words in the second set of text that are present in the first set of text divided by the total number of words in the second set of text.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
