The present invention relates to processing logged information and, more particularly, to efficiently searching through large volumes of logged data.
Computer systems record event information in the form of logs, with events such as system faults, failures, crashes, recovery, intrusion attempts, and normal operating events all producing log information. When an event occurs, logs are generated to indicate what happened before, during, and after the event. Computer forensics experts can then use the logs to investigate the causes of, and solutions to, problems in the system. However, such work can be very time-consuming, particularly in large, enterprise systems, where logs can be particularly voluminous.
Existing log pattern mining approaches take system logs as input and use data mining algorithms to discover patterns in the logs. Those patterns are then used as models for anomaly detection and system performance prediction. However, such approaches do not generally provide system event search capabilities.
A method for event detection and correction includes determining a log pattern for a received event. The log pattern is translated to an event search query. The event search query is weighted according to discriminative dimensions using term-frequency inverse-document-frequency. The event search query is matched to one or more known events. A corrective action is automatically performed based on a solution associated with the one or more known events.
A system for event detection and correction includes a pattern mining module configured to determine a log pattern for a received event. An event query module is configured to translate the log pattern to an event search query and to weight the event search query according to discriminative dimensions using term-frequency inverse-document-frequency. A search module includes a processor configured to match the event search query to one or more known events. A correction module is configured to automatically perform a corrective action based on a solution associated with the one or more known events.
These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
The disclosure will provide details in the following description of preferred embodiments with reference to the following figures wherein:
Embodiments of the present invention provide efficient query and retrieval of log information based on log pattern analytics and system event search. The present embodiments provide a translation between the logic of log pattern mining and the processes of the system event search, with results being rendered in a user-friendly format. The present embodiments can thereby automatically detect, diagnose, and correct system faults, providing an improvement to the technical fields of log analysis and automated system management.
Referring now in detail to the figures in which like numerals represent the same or similar elements and initially to
Block 104 mines patterns in a set of unknown events. For example, during runtime, new sets of logs are constantly being generated and processed. Block 104 identifies patterns within these new events using log analytics. The patterns that are found can include, for example, sequential patterns, periodic patterns, and correlation patterns.
Block 106 translates the output of the log analysis to an event search query. Block 108 performs an event search, matching the event search query to the database of signatures. The event search weights the importance of different dimensions in the vector representation of the logs according to, e.g., a term frequency, inverse document frequency (TF-IDF) scheme. Block 108 determines the similarity between events in the multi-dimensional space using any appropriate metric, such as the cosine similarity. In general, only some of the dimensions in the vector representation of the logs will be truly discriminative. The weighting identifies the useful dimensions by putting higher weights on those dimensions that are more discriminative.
The operation of block 108 includes a first step of term frequency normalization. Given log patterns that belong to a given pattern category, per-pattern counts are normalized by the total number of counts for the category. For example, if there are four patterns (P1, P2, P3, P4), and a particular event matches the patterns with counts of 1, 2, 3, and 4, respectively, then the vector representation of the event can be <1, 2, 3, 4>. After normalization, the vector representation becomes <0.1, 0.2, 0.3, 0.4>.
Inverse document frequency weights are then calculated based on the vector representation of the known events. If N is the number of known events, then xi is the vector representation of a system event I, and xi(j) is the value of xi in its jth dimension. The weight of the jth dimension is determined as:
These weights are then multiplied against the vector representation of the event, with the values of the vector being weighted as xi′(j)=xi(j)idf(j). The TF-IDF weighted vector representation of each known event is stored and is used for system event search queries.
Once one or more similar events have been determined by block 108, the results are represented as objects that include, for example, a relevance field, a data label field, and an “other” field. The relevance field provides a key/value tuple, where the key is an identifier of a known event and the value is the similarity between the search query and the known event, with a value of zero representing the minimum similarity and a value of one representing the maximum similarity. The data label field indicates the event identifier of the unknown event that makes up the search query. The “other” field records the signature of the unknown event. All matched results are stored and can be used for, e.g., ranking, filtering, visualization, and other appropriate actions.
Referring now to
Block 206 translates the log patterns that have been discovered by block 204 into search queries suitable for event searching and block 208 uses the search queries to generate event signatures. The event signatures can be a vector representation of the events in an appropriate space. Block 210 stores the generated event signatures in a signature database to be used for subsequent event search queries. The signatures can be stored as objects having keys that are pattern identifiers and values that are the signature values. These objects are stored in an appropriate database, with event signatures being indexed for high-efficiency searches.
Referring now to
In some embodiments, blocks 104/204 can identify both text and time-series based log patterns using analytical techniques to discover a variety of text-based patterns such as, e.g., schema patterns, sequential patterns, and periodic patterns. Time-series based log patterns can be mined using analytical techniques to discover time-series based patterns such as correlation patterns and burst patterns. Thus, a set of log patterns with a variety of pattern identifiers that appear in the known events, as well as the occurrence frequency of the patterns in the known events, can be obtained.
In other embodiments, block 104 accepts user controls regarding how input logs for known events can be accessed, the details of the signature database, the types and categories of log patterns to extract, and the type of request.
Referring now to
Referring now to
Block 504 then performs a search of a known events database as described above to identify one or more matching events. These matching events are tagged in the manner above to describe what type of event they represent. Block 506 then identifies a solution corresponding to the matched events. This solution can be stored with the matching events in the events database or can, alternatively, be stored in an external database that can be searched for particular solution information.
Block 508 then automatically implements the solution to repair or mitigate the underlying problem. Exemplary corrective actions include changing a security setting for an application or hardware component, halting and/or restarting an application, halting and/or rebooting a hardware component, changing an environmental condition, changing a network interface's status or settings, etc. In this manner, embodiments of the present invention can automatically respond to conditions and implement predetermined solutions to existing or emerging problems. This provides a substantial improvement in automated system monitoring and administration by efficiently accessing solution information, making it possible to respond to developing conditions more rapidly than would be possible by a human operator.
Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.
Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The medium may include a computer-readable storage medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk, etc.
Each computer program may be tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.
A data processing system suitable for storing and/or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code to reduce the number of times code is retrieved from bulk storage during execution. Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
Referring now to
The memory 604 stores a known event database that includes event signatures for a set of previously encountered events. The memory 604 also stores event logs 610. The event logs 610 are collected by one or more logging applications or hardware components that generate information in a continuous, periodic, or sporadic manner regarding monitored software or hardware components. This information can be generated locally to the system 600 or can, alternatively, be collected at other systems and then transmitted to the system 600 via the network interface 606.
An event signature module 612 generates event signatures from event logs 610 to be stored in the known event database 608. During runtime, the event signature module 612 generates event signatures for incoming event logs to be used to match against event signatures that are already in the known event database 608. A pattern mining module 614 performs analysis of the logs to determine log patterns that the event signature module uses to create the event signatures.
As new events arrive, event query module 616 takes the output of the pattern mining module 614 to generate search queries that represent the new patterns. Search module 618 then uses the search query to compare the new event to known events in the known event database 608. If the search module 618 finds a match (e.g., if a similarity score between the new event and a known event is above a threshold according to an appropriate similarity metric), then block 620 searches for a solution that matches the event. Correction module 622 then automatically performs a corrective action dictated by the solution module 620 to correct or mitigate a problem that caused the new event.
Referring now to
A first storage device 722 and a second storage device 724 are operatively coupled to system bus 702 by the I/O adapter 720. The storage devices 722 and 724 can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, and so forth. The storage devices 722 and 724 can be the same type of storage device or different types of storage devices.
A speaker 732 is operatively coupled to system bus 702 by the sound adapter 730. A transceiver 742 is operatively coupled to system bus 702 by network adapter 740. A display device 762 is operatively coupled to system bus 702 by display adapter 760.
A first user input device 752, a second user input device 754, and a third user input device 756 are operatively coupled to system bus 702 by user interface adapter 750. The user input devices 752, 754, and 756 can be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present principles. The user input devices 752, 754, and 756 can be the same type of user input device or different types of user input devices. The user input devices 752, 754, and 756 are used to input and output information to and from system 700.
Of course, the processing system 700 may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements. For example, various other input devices and/or output devices can be included in processing system 700, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized as readily appreciated by one of ordinary skill in the art. These and other variations of the processing system 700 are readily contemplated by one of ordinary skill in the art given the teachings of the present principles provided herein.
The foregoing is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the present invention and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.
This application claims priority to U.S. Provisional Patent Application No. 62/594,243, filed on Dec. 4, 2017, and to U.S. Patent Provisional Application No. 62,608,631, filed on Dec. 21, 2017, incorporated herein by reference herein its entirety. This application is furthermore related to U.S. patent application Ser. No. 16/203,008, filed on Nov. 28, 2018, incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
62608631 | Dec 2017 | US | |
62594243 | Dec 2017 | US |