Patent Application
20230186022
The present disclosure relates to natural language processing, and in particular relates to processing of a corpus of documents having both natural language elements and computer language elements.
In Natural Language Processing (NLP), documents can be classified by finding specific sets of words in the document, and based on the sets of words found, the type of document may be identified. This is sometimes referred to as topic modelling, where the term ‘topic’ represents a set of words. In topic modeling, a model may be trained to automatically discover topics appearing in documents.
Latent Dirichlet Allocation (LDA) is one example of a topic modelling technique. This technique assumes that the distribution of topics in a document and the distribution of words in a topic are both Dirichlet distributions. LDA assigns words to topics, and if the collection of documents is sufficiently large, LDA may discover topics based on the co-occurrence of individual terms/words.
LDA has been shown to provide good results when a single language appears in the documents, or where various natural languages are in the corpus of documents, but where the documents are translations of each other. However, certain corpuses may contain other information which may not be effectively modelled using LDA. For example, technology blogs, programming groups, computer help sites, among others, may contain both computer language such as source code, log data, and the like, mixed with natural language elements. Current LDA techniques do not provide good associations between such computer language terms and natural language terms.
The present disclosure will be better understood with reference to the drawings, in which:
The present disclosure provides a method at a computing device comprising: mapping, within a corpus of documents having both natural language terms and computer language terms, each term as a natural language term or a computer language term, thereby creating mapped terms; and applying at least one Latent Dirichlet Allocation (LDA) model to the mapped terms to create topics that correlate the natural language terms and computer language terms.
The present disclosure further provides a computing device comprising: a processor; and a communications subsystem, wherein the computing device is configured to: map, within a corpus of documents having both natural language terms and computer language terms, each term as a natural language term or a computer language term, thereby creating mapped terms; and apply at least one Latent Dirichlet Allocation (LDA) model to the mapped terms to create topics that correlate the natural language terms and computer language terms.
The present disclosure further provides a computer-readable medium for storing instruction code, which, when executed by a processor of a computing device cause the computing device to: map, within a corpus of documents having both natural language terms and computer language terms, each term as a natural language term or a computer language term, thereby creating mapped terms; and apply at least one Latent Dirichlet Allocation (LDA) model to the mapped terms to create topics that correlate the natural language terms and computer language terms.
Computer language is often used with natural language in documents pertaining to computer issues. For example, software logs are a valuable resource in the diagnosis of failures in software systems. When end users describe software failures, often they may combine natural language such as English with computer logs. Examples of channels in which end users describe technical problems with a combination of natural language and logs may include tickets in information technology (IT) ticket systems; online chat with customer technical support; posts in technical support forums; among others.
Further, computer language is not limited to logs and in some cases could include source code, error screen or error message information, among other options. Logs are used as a source of computer language within documents such as support threads in the examples below merely for illustration, and other forms of computer language could equally be used.
Thus, the term “computer language” as used herein refers to program-generated text that relates to operating or debugging a computer, including source code and log entries, among other options. For example, such computer language may include output from a log, an error report generated by a computer, a screen dump, or other similar program or computer generated output.
Latent associations may exist between pieces of text written in natural language and certain log excerpts in support thread datasets. Associations may be based on an original post in which a user may describe a problem, or in a subsequent post in response to a request for the log excerpts. For example, there may be multiple reports of a specific operating system crash that is associated with the log excerpt “VIDEO.TDR.FAILURE (igdkmd64.sys)”. Discovering such latent associations enables the construction of a knowledge base that links problem discussions to log excerpts that are typically associated with these discussions.
Ultimately, such a knowledge base speeds up failure debugging and technical support service quality.
Therefore, in accordance with the embodiments of the present disclosure, systems and methods are provided to address how to find associations between natural language and computer language.
In one embodiment, the problem of establishing connections between natural language terms and computer language terms such as text within logs can be seen as a problem of establishing connections between pieces of texts that are written in different languages. More specifically, computer language such as system logs may be produced by a given tool and can therefore be seen as a semi-informal language that has its own syntax, vocabulary and semantics.
LDA is typically used on a corpus of documents which is monolingual. As used herein, a corpus of documents is a collection of documents, where each document is a collection of words/terms. As used herein, a “term” may consist of one word or a group of words.
Reference is now made to
Each document is provided to the standard LDA pipeline 120. The text of each document may then be subjected to preprocessing at block 130. Preprocessing may include tokenization, lemmatization, stemming, and removal of stopwords, among other preprocessing steps. Thus, at block 130 the preprocessing may make any pluralized words singular; convert any verb to the present tense; remove certain words such as “the”, “a”, “and”, among others; words in the third person may be changed to the first person; all words may be converted to lowercase; punctuation may be removed; words may be reduced to their root form; very common words may be pruned; hypertext markup language (html) tags may be removed; special characters may be removed; among other options for preprocessing.
The preprocessed dictionary may then be provided to a vector space block 140. Vector space block 140 may create vectors for various topics. For example, referring to
Referring again to
The process then proceeds to block 160 in which an evaluation of the LDA model may be made. The evaluation may be based on one or more criteria to see how well the model classifies documents.
Based on
However, the model of
In this regard, three embodiments are described below. In a first embodiment, a prefix may be added to words in the document to distinguish between natural language in computer language. In a second embodiment, natural language text can be split from computer language text, and each can be processed by an LDA model. The results can be combined, and the combined results can then be further processed by another LDA model. In a third embodiment, a multilingual LDA model that is adapted to utilize a corpus of documents which are translations of each other, can be enhanced to operate with computer language and natural language. Each are described below.
Standard LDA models are monolingual, which means that they cannot align topics across languages. In some cases, standard LDA models can still produce topics containing words for multiple languages if run on multiple input documents. Such multilingual topics can uncover relationships between words from different languages. However, in cases where there are natural language and computer language terms that are loosely associated, the standard LDA models do not produce good results.
Therefore, in one embodiment of the present disclosure, a prefix may be added to words/terms in the original documents, such that natural language text and computer language text can be differentiated.
Reference is now made to
For example, in one case the documents may be sourced from a forum such as a community question-answer forum. Such forum may, for example, be organized into various sections and subsections such as operating system support sections which may include operating system crashes in one subsection and other issues with the operating system in other subsections.
In such forum, users may ask technical questions by starting a thread on a given forum subsection. Any user can respond to a question by providing one or more replies. In a reply, form members typically asked for question clarifications, suggest that the original poster run diagnostic software tools, examine the logs produced by the diagnostic software tools, and offer potential solutions to the problem. Thus, a post may be either a question or a reply to the question. A thread may contain natural language text such as English text and may further contain computer language text such as system logs. However, in some cases, the post may not contain any system logs.
In some embodiments, it may be beneficial to limit input documents to a single type of computer language. Specifically, various diagnostic tools, logs, source code, among others, may be found in different threads, and each may be considered to be a different language. For example, logs produced by each log tool may differ in terms of structure, vocabulary and grammar. Therefore, each log tool may produce logs in a different language and the embodiments of the present disclosure may, in some cases, choose threads or posts which have the same log structure and semantics to be part of a corpus of documents in order to create the topics for the documents within that corpus of documents.
In the embodiments of the present disclosure, the systems and methods uncover associations between natural language terms and computer language terms, and therefore utilizing the same computer language between posts may be beneficial to create such associations.
However, in some cases logs or other computer language text from different sources may be considered to be similar enough to combine in a single corpus of documents.
In other cases, other criteria may be applied to create the corpus of documents. For example, in some cases each document in the corpus of documents may be required to have both computer language terms and natural language terms.
In some cases, documents that are monolingual, and have only one of the natural language terms or computer language terms, may be included. For example, this may occur based on the location of the documents within certain sections of computer help websites in some cases.
In other cases, other filters may be applied to create the corpus of documents forming input documents 310.
Other options are possible for the input documents 310.
In the embodiment of
In one example, regular expressions may be used to detect the presence of boundaries of logs in a thread. For example, a logging tool may have standardized text at the beginning of a log that can be used as a differentiating factor between in the text and logs. In this case, the mapping function may map the words/terms before the log boundary as natural language text and the words/terms after the boundary as computer language text.
Similarly, in some cases a boundary may exist at the end of a system log, and the mapping function 320 may map words/terms before the ending boundary (but after the beginning boundary) as computer language terms and words/terms after the ending boundary as natural language terms.
For example, a tool such as MiniToolBox™ by Farbar may create logs that start with “MiniToolBox by Farbar Version: 18-01-2012, Ran by **** on Apr. 04, 2020 at 04:10:55” and this could be used as the beginning boundary. In other cases, only sections of the log files may be used, and may contain headers such as “===================== FF Proxy Settings: ========================”, or the like. The end boundary may be similarly be distinctive text based on the logging program.
In other cases, a dictionary of terms which can be found in the computer log could be created to allow mapping function 320 to distinguish between natural language text and computer language text. In some cases the context of the word may be analyzed in cases were a computer language term and a natural language term could be the same.
Other options are possible.
Once the terms are distinguished, mapping function 320 may add a prefix such as “text_” before any natural language word/term and “log_” before any computer language word/term. Thus, the enriched LDA model of
The output from mapping function 320 may then be provided to a standard LDA model 330. The functionality from, for example, standard LDA pipeline 120 of
Thus, in some cases, preprocessing may occur on the words from the mapping function 320. In some embodiments, the preprocessing may be done on all words from the mapping function 320. In some cases, the preprocessing may only occur on natural language words and no preprocessing would occur for the computer language words.
In some embodiments, some preprocessing, such as tokenization, lemmatization, stemming, and removal of stopwords could occur prior to adding the prefix at the mapping function 320. In some embodiments, the preprocessing, such as tokenization, lemmatization, stemming, and removal of stopwords could occur at the LDA model 330, where the LDA model 330 is configured to consider portions after the prefix for such preprocessing. Again, the LDA model 330 could ignore any terms having a computer language prefix in some cases.
Therefore, the embodiment of
The embodiment of
In some cases, the embodiment of
In the embodiment of
The words/terms within the corpus of input documents 410 could be distinguished between computer language text and natural language text. This may be done, for example, by using a mapping function 412, which may be similar to mapping function 320 from the embodiment of
Next, two topic models are used. A first is for natural language text only and a second is for computer language text only. Specifically, as seen in
The processing at standard LDA pipeline 420 and the processing at standard LDA pipeline 430 may be similar to the processing performed in the embodiment of
The output from the standard LDA pipeline 420 and the output from standard LDA pipeline 430 are then provided to a document summarization block 440. At block 440 the contents of the original document are replaced with the natural language topics and the computer language topics. More specifically, for each topic model, a document specific topic is first identified according to topic probabilities.
The top number of terms by term probability may then be extracted (for example for display or mnemonic purposes). Extraction may, however, in some cases not occur and the complete set of terms may be used. If extraction occurs, the actual number of terms may be chosen based on design considerations and may in some cases be fixed. For example, the top ten terms may be extracted in some cases. However, the embodiments of the present disclosure are not limited to extracting ten terms, and a fewer or a greater number of terms could be extracted in some cases. Further, in some cases the number of terms extracted may be variable based on factors within the system.
Further, to identify the origin of each term, topic indexes may be appended at the end of each term in some embodiments. For example, a term may be composed as “LOG:stamp:TOPIC:399”. This term indicates that the word “stamp” comes from a log file and is associated with topic 399.
At the document summarization block 440, the top terms extracted from both from LDA pipeline 420 and LDA pipeline 430 are then merged by document, creating a corpus with topics 450. In some cases, corpus with topics 450 has the same corpus structure as input documents 410, but has topics added to such structure.
Corpus with topics 450 may now act as an input to a third LDA topic model, namely standard LDA pipeline 460.
The output from LDA pipeline 460 provides correlated topics for the original input documents.
Based on this, two layers in the model of
A second layer 480 processes the corpus of documents having topics embedded therein to produce correlated topics for the original input documents.
Therefore, the embodiment of
Multilingual LDA models are designed to find topic alignments across documents written in different languages. However, such multilingual models are generally a directed to documents that are direct translations of one another.
In the context of documents having both natural language and computer language elements, the computer language elements are often loosely related to the natural language components, as opposed to being direct translations, and thus the multilingual model may be unsuitable.
In this regard, in a further embodiment of the present disclosure, an existing bilingual candidate model is adapted for computer language text. Reference is now made to
In particular, the embodiment of
Thus, in the embodiment of
In some cases mapping may occur to create the computer language and natural language portions.
The output from the loosely parallel corpus 520 is then provided to a standard LDA model. In particular, preprocessing occurs at block 560 and could be similar to the preprocessing done at block 130 in
Vector spaces may be created at block 562, which may be done in a similar manner to the creation of vector spaces at block 140 in
A bilingual LDA then is processed at block 564 with the division of the text and logs.
An evaluation may then occur at block 566. The evaluation at block 566 may be similar to the evaluation performed at block 160 in
Thus, the embodiment of
LDA models need to produce a coherent set of topics. In other words, the LDA model needs to produce a set of words describing a topic that is semantically connected. In this regard, a topic coherence metric may be used to determine the degree of semantic similarity between high scoring words in a topic.
A further evaluation of quality may be the level of inferential power achieved by the candidate model. Specifically, LDA models are useful when they can correctly guess one or more of the higher weighted topics of a new, unseen document. Specifically, in LDA each document will typically have a probability or weighting for the topics in the corpus associated with that document, and the higher probability or weighting scores for topics in the document will be indicative of the topics of the document. In one case a document having “mostly” topic 1 (of say, ten topics in the corpus) might have a topic assignment of (t1:90%, t2:8%, t2:1.9%, t4...t10: less than 0.1%). Thus the level of inferential power can be based on whether the model can correctly guess the one or more most highly weighted topics. Therefore, equality metric may be a “perplexity” score, where the model can be evaluated on how perplexed it becomes when it encounters words of a new, unseen document.
A further evaluation of quality maybe the text-log (natural-computer language) alignment achieved by the candidate models. In particular, LDA maybe used to produce topics that contain both natural language words and computer language words. In this regard, a quality metric may include the level of alignment.
Hyperparameter tuning, for example using the techniques described by J. Ansel et al, “An extensible framework for pro-gram autotuning”, proceedings of the 23rd International Conference on Parallel Architectures and Compilation, PACT ′14, Association for Computing Machinery, New York, NY, USA, 2014, p. 303-316 may be used to tune the LDA models.
Utilizing the above evaluation criteria, the embodiments of
Further, with regards to perplexity, the embodiment of
Comparing the embodiments of
Each was able to align natural language and computer language terms in a corpus of documents and any could be used as practical implementations for linking such loosely associated terms.
Therefore, the embodiments of the present disclosure provide for the linking natural language (e.g., English) to computer language (e.g., system logs). This may be applied to technical support threads in some cases.
Uncovering such links may enable the construction of a knowledge base that connects technical problem discussions to log or computer language excerpts that are often associated alongside those discussions.
Various embodiments of LDA models used to create such links are therefore provided.
The above models may be implemented using any computing device or combination of computing devices. One simplified diagram of a computing device is shown with regard to
In
Processor 620 is configured to execute programmable logic, which may be stored, along with data, on device 610, and shown in the example of
Alternatively, or in addition to memory 632, device 610 may access data or programmable logic from an external storage medium, for example through communications subsystem 630.
Communications between the various elements of device 610 may be through an internal bus 650 in one embodiment. However, other forms of communication are possible.
The embodiments described herein are examples of structures, systems or methods having elements corresponding to elements of the techniques of this application. This written description may enable those skilled in the art to make and use embodiments having alternative elements that likewise correspond to the elements of the techniques of this application. The intended scope of the techniques of this application thus includes other structures, systems or methods that do not differ from the techniques of this application as described herein, and further includes other structures, systems, or methods with insubstantial differences from the techniques of this application as described herein.
While operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be employed. Moreover, the separation of various system components in the implementation descried above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Also, techniques, systems, subsystems, and methods described and illustrated in the various implementations as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made.
While the above detailed description has shown, described, and pointed out the fundamental novel features of the disclosure as applied to various implementations, it will be understood that various omissions, substitutions, and changes in the form and details of the system illustrated may be made by those skilled in the art. In addition, the order of method steps are not implied by the order they appear in the claims.
When messages are sent to/from an electronic device, such operations may not be immediate or from the server directly. They may be synchronously or asynchronously delivered, from a server or other computing system infrastructure supporting the devices/methods/systems described herein. The foregoing steps may include, in whole or in part, synchronous/asynchronous communications to/from the device/infrastructure. Moreover, communication from the electronic device may be to one or more endpoints on a network. These endpoints may be serviced by a server, a distributed computing system, a stream processor, etc. Content Delivery Networks (CDNs) may also provide may provide communication to an electronic device. For example, rather than a typical server response, the server may also provision or indicate a data for content delivery network (CDN) to await download by the electronic device at a later time, such as a subsequent activity of electronic device. Thus, data may be sent directly from the server, or other infrastructure, such as a distributed infrastructure, or a CDN, as part of or separate from the system.
Typically, storage mediums can include any or some combination of the following: a semiconductor memory device such as a dynamic or static random access memory (a DRAM or SRAM), an erasable and programmable read-only memory (EPROM), an electrically erasable and programmable read-only memory (EEPROM) and flash memory; a magnetic disk such as a fixed, floppy and removable disk; another magnetic medium including tape; an optical medium such as a compact disk (CD) or a digital video disk (DVD); or another type of storage device. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly a plurality of nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.
In the foregoing description, numerous details are set forth to provide an understanding of the subject disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details discussed above. It is intended that the appended claims cover such modifications and variations.
