INFORMATION EXTRACTION AND ANNOTATION SYSTEMS AND METHODS FOR DOCUMENTS

FIELD OF THE INVENTION

The present technology may be generally described as systems and methods for extracting information from documents and determining annotation instances via analysis of the extracted information using various methods such as rule-based methods and feature-based methods.

BACKGROUND

In traditional approaches to information extraction, both engineering and domain knowledge are required to build an extractor that is used to extract information from documents. An extractor engineer using a complex development environment creates extractors manually. Especially for simple in-text fields (which constitute a majority of fields in most extraction projects), the generation of effective extraction rules seems possible for a domain expert without engineering knowledge. However, the complexity of the development environment currently requires extractor engineering knowledge that most domain experts do not possess.

SUMMARY

According to some embodiments, the present technology is directed to a method that includes: (a) receiving training documents, the training documents having annotated words; (b) identifying a predetermined number of characters preceding and following each annotated word for each of the training documents to determine a context for each of the annotated words; (c) performing an alignment of an annotated word and its context with characters in the target document; (d) identifying common sequences between the target document and training corpus; based upon the alignment; and (e) assigning an annotation to a word in the target document when a common sequence between the target document and training corpus is found, the annotation comprising an annotation of an annotated word of the training corpus included in the common sequence.

According to some embodiments, the present technology is directed to a method that includes: (a) receiving training documents, the training documents having annotated words; (b) identifying a predetermined number of characters preceding and following each annotated word for each of the training documents to determine a context for each of the annotated words; (c) dividing the target document into n-gram strings; (d) identifying overlapping sequences of characters between the n-gram strings of the target document and the training documents; and (e) assigning an annotation to each word in the target document when the overlapping sequences between the n-gram strings of the target document and the training documents meet or exceed a threshold.

According to some embodiments, the present technology is directed to a method that includes: (a) receiving training documents, the training documents having annotated words; (b) for each annotated word, determining a distance between an annotated word and one or more document features; (c) calculating an averaged location for each annotated word, the averaged location defining an averaged placement of an annotated word within a training document; (d) for each word of a target document, determining a distance between an annotated word and one or more document features; (e) calculating an averaged location for each annotated word, the averaged location defining an averaged placement of an annotated word within a target document; (f) identifying matches between the distance and averaged location of words of the target document and the distances and averaged location of at least one of the annotated words of the training documents; and (g) assigning an annotation to a word of the target document that matches the at least one of the annotated words of the training documents.

According to some embodiments, the present technology is directed to a system that includes: (a) a processor; and (b) logic encoded in one or more tangible media for execution by the processor, the logic when executed by the processor causing the system to: (i) receive training documents, the training documents having annotated words; (ii) identify a predetermined number of characters preceding and following each annotated word for each of the training documents to determine a context for each of the annotated words; (iii) perform an alignment of an annotated word and its context with characters in the target document; (iv) assign an annotation to a word in the target document when a common sequence between the target document and training corpus is found, the annotation comprising an annotation of an annotated word of the training corpus included in the common sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain embodiments of the present technology are illustrated by the accompanying figures. It will be understood that the figures are not necessarily to scale and that details not necessary for an understanding of the technology or that render other details difficult to perceive may be omitted. It will be understood that the technology is not necessarily limited to the particular embodiments illustrated herein.

FIG. 1A is schematic diagram of an exemplary information extraction system, constructed in accordance with the present technology;

FIG. 1B is schematic diagram of an exemplary annotation system, constructed in accordance with the present technology

FIG. 2 illustrates an exemplary annotated document;

FIG. 3 illustrates the creation of exemplary word vectors;

FIG. 4 illustrates the creation of exemplary rules from word vectors;

FIG. 5A-C each illustrate various embodiments of graphical user interfaces for annotating documents;

FIG. 6 illustrates a context analysis process for selected strings having similar annotated fields;

FIG. 7 illustrates a process of extracting and normalizing common subsequences;

FIG. 8 illustrates a process of determining a frequency of occurrence of subsequences;

FIG. 9A-B collectively illustrate a method of transforming context instances (e.g., subsequences) into an n-gram representation that can be utilized by a classifier model to annotate target documents;

FIG. 10 is an exemplary set of executable instructions for calculating the contiguity of words within a string;

FIG. 11 is an exemplary set of executable instructions for calculating the mean length of words within a string;

FIG. 12 is an exemplary set of executable instructions for calculating a longest contiguous subsequence between a pair of strings;

FIG. 13 is an exemplary set of executable instructions for creating a feature set from suggested sequences generated by the instructions of FIG. 12;

FIG. 14 illustrates a method having steps that are executed in accordance with the present disclosure;

FIG. 15 illustrates another exemplary method having steps that are executed in accordance with the present disclosure;

FIG. 16 is a flowchart of an exemplary method for automatically annotating documents;

FIG. 17 is a flowchart of another exemplary method for automatically annotating documents;

FIG. 18 is a flowchart of an additional exemplary method for automatically annotating documents;

FIG. 19A illustrates the determination of average distances for annotated fields (or annotatable fields) in a document, their position being relative to section beginnings or document number;

FIG. 19B illustrates the determination of average distances for annotated fields (or annotatable fields) in a document, their position being relative to preceding subsection locations;

FIG. 19C also illustrates the determination of average distances for annotated fields (or annotatable fields) in a document, their position being relative to preceding subsection locations; and

FIG. 20 illustrates an exemplary computing system that may be used to implement embodiments according to the present technology.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

While this technology is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail several specific embodiments with the understanding that the present disclosure is to be considered as an exemplification of the principles of the technology and is not intended to limit the technology to the embodiments illustrated.

It will be understood that like or analogous elements and/or components, referred to herein, may be identified throughout the drawings with like reference characters. It will be further understood that several of the figures are merely schematic representations of the present technology. As such, some of the components may have been distorted from their actual scale for pictorial clarity.

Generally speaking, the present technology provides systems and methods for extracting information from documents, and in some instances, unstructured and semi-structured documents such as contracts. That is, a contract may include general sections such as price, duties, warranties, and so forth. In other instances, informal contracts, such as letters of intent, may include freeform documents having information that is not arranged in any particular manner.

The present technology makes extraction of in-text fields within documents more scalable by implementing extraction methods that do not require specific programming/engineering skills, but only domain knowledge. The present technology is more scalable in comparison to manual processes where both engineering and domain knowledge are required to build an extractor that is used to extract information from documents. That is, a domain expert initially examines and annotates the documents, formulates general extraction rules, and an extractor engineer translates those rules into executable extractors that are used to process documents.

Manual approaches focus on creating optimal regular expressions from a set of examples. In contrast, the present technology may be configured to identify discriminative sequences within documents and use these discriminative sequences to develop both rule-based extractors and feature-based extractors. The terms “discriminative sequence” is defined as a sequence of words within a document that are categorized as having a particular domain. For example, a discriminative sequence that indicates the start date of a contract may include the terms ‘start,’ ‘begin,’ ‘commence,’ or other similar terms that are spaced apart from a date within a given character proximity. In one example, the term ‘start’ is found within three words of a date, and is thus characterized as a discriminative sequence for a contract start field.

While optimal regular expressions try to generalize the pattern found, such methods have limitations that render the use of optimal regular expression less desirable in real-world extraction environments. Finding discriminative sequences is less ambitious in regard to generalization and therefore more suitable for real-world extraction environments.

In some instances, the present technology may utilize rule-based methods. In these methods, the user provides a sufficient number of annotations for a specified field within a set of training documents. The present technology identifies discriminative sequences in the context of these annotations. Based on these sequences rule suggestions are generated and provided to the user. The user may select or modify the suggested extraction rules and may save the extraction rules in a database. These rules (e.g., extractors) are exported into an extraction system and utilized to process target documents. This process creates an abstraction of the engineering function that would normally be performed by an extractor engineer. Thus, no programming knowledge is required by the domain expert in order to build an extractor.

According to some embodiments, the present technology may utilize sequence alignment to identify discriminative sequences in the context of a specified field. More specifically, similar sequences found in a plurality of annotated documents may be aligned with one another. Using alignment between these similar sequences, in context, discriminative sequences may be determined.

In some instances, the present technology may utilize a feature-based method for processing documents, where alignments in context are used to build features, rather than rules. These features can be used to train a classification engine. Based on a number of context-dependent features, a classification engine may decide whether a potential match is in fact the information that should be extracted.

The present technology may utilize a longest common subsequence algorithm to extract discriminative contextual features for extraction. Additionally, this longest common subsequence analysis is fine-tuned using a contiguity analysis. A greedy contiguity heuristic may be utilized to quantify aspects of contiguity within aligned sequences by analyzing the number of match segments, number of gap segments, and variance in segment length.

These and other advantages of the present technology will be described in greater detail below with reference to the drawings.

FIG. 1A illustrates an exemplary information extraction system (hereinafter “system 100”) that generates and applies rule-based extractors as well as feature-based vectors. Generally, the system 100 is shown as comprising a UI (User Interface) module 105 that generates graphical user interfaces that are utilized by document annotators to upload or annotate original documents within the system 100. In some instances, the system 100 may include a base annotation module 105A that is utilized to generate annotated training documents 110. Representative UIs are described in greater detail with reference to FIGS. 5A-C, which will be described in greater detail below.

In some instances, the system 100 is configured to generate extractor rules that can be utilized by a rule-based annotator to annotate target documents. In other embodiments, the system 100 may be configured to generate feature-based vectors that can be utilized to train a classifier model annotator to annotate target documents. These various methodologies for annotating target documents will be described in greater detail below with reference to FIG. 1B.

A document annotator, such as a document domain expert may utilize various UIs to annotate the original documents to produce annotated documents. A document annotator may utilize a set of annotation tools, such as a common field lists. Again, exemplary annotation tools are illustrated and described in greater detail below with reference to FIGS. 5A-C. It will be understood that the more annotated training documents available to the system, the more robust and comprehensive of a rule set may be made available to the system 100.

Generally, annotation of a document may include associating a word, date, or other group of characters (e.g., identifiable strings) with a particular field. In some instances, the annotations may consider these identifiable strings within their context relevant to the document. For example, the word ‘indemnification’ may be annotated as belonging to a field “Warranty and Indemnification.” The word indemnification may be confirmed as belonging to the Warranty and Indemnification field because the word indemnification was found within a certain distance from the word “warranty.”

FIG. 2 illustrates the annotation of a contract document 200. In this instance, a document annotator has annotated a first date 205 as being associated with a Start Date field, as well as a second date 210, which is also associated with another Start Date field. In this example, the Start Date is a defined field that refers to dates within a document that directly relate to a start date for beginning various types of work or other duties/performances by parties to the contract. Many standard fields may be utilized, as well as annotator-defined fields.

The system 100 may employ a context and contiguity analysis module 115 to evaluate these annotated documents, specifically to generate extractor rules 130 and feature based vectors 135. For brevity, the context and contiguity analysis module 115 will hereinafter be referred to as the “analysis module 115”. The analysis module 115 determines discriminative sequences within strings in documents that comprise annotated fields. In some instances, determining discriminative sequences comprises locating (or closely corresponding) words or groups of characters that are proximate to the field date. In some instances the analysis module 115 evaluates words appearing before and after the annotated dates.

It will be understood that a discriminative sequence may include a single word or a group of characters, or may also include a group of words, numbers, symbols, or other similar textual structures. The discriminative sequence is used as a signal or identifier that provides a context for the annotated field with which they are associated. For example, the word ‘commencing’ within three words of an annotated field of a date is discriminative inasmuch as the word ‘commencing’ indicates that the annotated date is a performance start date.

After determining discriminative sequences, a rule-based extractor 120 may utilize the discriminative sequences to generate proposed extraction rules. An exemplary rule is shown in FIG. 4. The rule-based extractor 120 may provide the document annotator with the proposed extraction rule in an algorithmic form or natural language format.

The document annotator may approve the proposed extraction rule, reject the proposed extraction rule, and/or modify the proposed extraction rule. Once approved (either directly or after modification), the rule-based extractor 120 exports the proposed extraction rule as an extractor rule 130 that may be stored in a datastore 140 or provided directly to an annotation system 145.

FIG. 3 illustrates the creation of two regular expression vectors 305 and 310. It is noteworthy that each of the two regular expression vectors 305 and 310 includes a number of slots that represent preceding vector segment (slot) a target annotation segment, and a succeeding vector segment, although other segments may also be included therein.

A regular expression vector may be translated by the rule-based extractor 120 into rules that may be utilized to create an extractor rule 130. FIG. 4 illustrates the generation of two rules. A first rule 405 corresponds to the regular expression vector 305 of FIG. 3. Thus, a Start Date regular expression rule would include a Slot 1 having the characteristics of [[start]ing], which includes each of the various contextual forms of the word ‘start.’ Slot 2 of the first rule 405 comprises the actual date located in the document, and Slot 3 includes the characteristics of [A[a[greement]]], which includes each of the various contextual forms of the word ‘agreement’ including uppercase and lowercase forms of the word. It will be understood that a slot may include a representation of various permutations of a word. For example, if the words ‘start’ and ‘starting’ were determined to be discriminative relative to annotated field dates, the [[start]ing] information in Slot 1 is a way of representing this permutation.

The first rule 405 can be expressed linguistically as, “for the various regular expressions of ‘start,’ which are followed by a Date, where the Date is followed by various regular expressions of ‘agreement,’ classify or annotate the Date as a Start Date.

Turning to FIG. 1B, the annotator system 145 may apply the extractor rule 130 against target documents 150 in order to automatically annotate the target documents 150 using a rule-based annotator 160. In some embodiments, the rule may be used to locate fields within documents and extract the contents thereof. The extractor system 145 may apply the extractor rule 130 against other target documents 150 to locate Start Date fields within the target documents 150.

Automatically annotated documents 165 may be generated by the annotator system 145 initially annotating documents with base annotations. For example, the documents may be processed to determine general fields such as names, dates, and so forth. Next, strings within the target documents 150 compared by the annotator system 145 to an applied rule. If the string corresponds to the rule, the date within the string may be automatically annotated as a Start Date field. In general, the automatically annotated documents 165 may be categorized by the annotator system 145 according to the annotations specified by the rule.

FIGS. 5A-C collectively illustrate various graphical user interfaces that are utilized by a document annotator to generate (annotate) training documents 110. FIG. 5A includes a UI 500 that comprises a list 505 of Extractor Fields that include the Extractor Fields available to the annotator. A training document 510 is loaded in to a workspace of the UI 500. Base annotations 515 and 520 are shown on the training document. These base annotations may be highlighted for the annotator using, for example, a highlighting color. It will be understood that a “base” annotation includes a basic determination that a group of characters or a word belongs to a particular field. An exemplary base annotation may include “date.” A base annotator (see FIG. 1) may process a document and locate and annotate all “date” fields within the document.

FIG. 5B illustrates a UI 525 that is similar to the UI 500 of FIG. 5A, illustrating the annotation of the target document, where a document has specifically annotated dates comprising a Start Date and an End Date. FIG. 5C illustrates the process of a base annotation being selected and assigned a specific annotation.

Turning back to FIGS. 1A and 1B, the base annotation module 105A may pre-process the original documents utilizing a set of base annotations, such as Date, Name, and so forth. Base annotation of a document allows the document annotator to quickly identify words/characters that have been flagged as potentially annotatable.

In the example shown in FIG. 7, the contextual words ‘begin’ and ‘begins’ are found to correspond between the two aligned strings. The proximity of the words ‘begin’ and ‘begins’ relative to their respective annotated fields may also be used as a discriminative factor. That is, it is reasonable to assume that a discriminative sequence may be positioned more closely to the annotated fields, compared to words that are more remote to the annotated fields, especially when the discriminative sequence includes general words such as ‘begin.’

More specifically, a pair-wise sequence alignment may be executed on all similar contexts within annotated documents. That is, the analysis module 115 may align a first string of a first date with a second string of a second date (as shown in FIG. 7), when the first and second dates have been annotated as comprising similar field types, such as Start Date. In other words, the first and second strings are aligned by the analysis module 115 because they have been annotated as belonging to the same field type.

According to some embodiments, after the documents have been annotated with base annotations, the analysis module 115 may be executed to extract a longest contiguous common subsequence (LCCS) from every possible pair of context strings found between an aligned pair of strings. In some instances, common subsequences are extracted from every possible pair of contexts, and then grouped by similarity.

The analysis module 115 initiates this LCCS process by determining a context for keywords within the aligned strings by examining characters or words that appear before or after the annotated words/characters. The analysis module 115 then cleans or normalizes subsequences by extracting all matching segments of length>n₁aggregated by similarity. FIGS. 6 and 7 collectively illustrate a process of normalizing aligned strings, which have been annotated as comprising a Start Date annotation.

According to some embodiments, the analysis module 115 then assigns a frequency to each LCCS. FIG. 8 illustrates the generation of frequency scores from the annotated and contextually analyzed strings shown in FIGS. 6 and 7. For example, the context analysis module 115 has determined that an LCCS of ‘start’ has been found within an annotated document at a frequency of one. Additionally, the analysis module 115 has determined that an LCCS of ‘begin’ has been found within an annotated document at a frequency of two (see FIG. 8).

It is noteworthy that the LCCS process can be utilized to create a ‘normalized’ representation of similar words that may have slight differences in spelling, such as when the words ‘begin’ and ‘begins.’ The context of these words within a string provides indications to the analysis module 115 that the words are synonymous. In some instances, the context is determined by determining the distance of the word from the annotated words/characters.

In FIG. 7, the analysis module 115 has determined that an LCCS of ‘begin’ exists relative to the words ‘begin’ and ‘begins.’ Moreover, the context analysis conducted by the analysis module 115 implies that the words ‘begin’ and ‘begins’ are similar concepts because of their relative positions compared to the annotated words/strings (e.g., the Start Date fields).

According to some embodiments, the analysis module 115 may be configured to assume that examples that were flagged by the base annotation module 105A, but not annotated by the document annotator as a specific annotated field, as negative instances. Conversely, the analysis module 115 considers annotations specifically defined by the document annotator as positive instances.

Instead of rule suggestions, features are created (e.g., feature representation) that are used to train a classifier model annotator 155 used by the annotator system 145. More specifically, feature-based extractor generator 125 may generate feature representations that are converted into a feature-based vector 135. The system 100, and specifically the feature-based extractor generator 125, creates a feature-based vector 135 that can be utilized by the classifier model annotator 155 to annotate target documents.

In some instances, features are mapped to a high-dimensional feature representation. Exemplary classifier models may comprise a support vector machine (SVM), a decision tree, or a naïve Bayes classifier, or other similar classifier that would be known to one of ordinary skill in the art with the present disclosure before them.

The analysis module 115 may utilize discriminative sequences to create a feature representation that is based on alignment percentage and that localizes and minimizes effects of OCR errors present in the target documents 150, if any. Additionally, the feature representation may be created in such a way that it is nuanced, allowing for weighting of parts of subsequences.

With regard to determining high-dimensional feature representations, instead of building rules using the most frequently identified sequences, frequent sequences serve as cluster centers. Once the sequences for the cluster centers have been identified, each sequence is transformed into n-grams using a windowing algorithm, which will be described in greater detail below. Resulting n-grams are used to define a set of features.

Each new aligned sequence of a context is split into n-grams and the distance (e.g., alignment coverage) between the different n-grams of the feature vector is computed. The resulting feature vector holds percentages of alignment coverage for each new context.

This information may be used by the annotator system 145 to train a classifier model annotator 155 using the annotated training data, expressed as feature-based vectors. For target documents 150, new context instances are first transformed to n-gram representation and then fed to the classifier model annotator 155. In some instances, the classifier model annotator 155 allows addition of additional meta-features beyond simple analysis of context. For example, distance between subsequence matches and instances, as well as other factors such as document section and the like. Annotated documents 165 may be generated by the extractor system 145 employing the classifier module annotator 155.

FIGS. 9A and 9B collectively illustrate a method of transforming context instances (e.g., subsequences) into an n-gram representation that can be utilized by a classifier model to classify target documents. FIG. 9A illustrates context instances extracted from documents and FIG. 9B illustrates the transformation of these context instances into an n-gram representation using a windowing algorithm.

The dynamic programming solution to the longest common subsequence processes described herein may be augmented by the use of a greedy contiguity heuristic. This heuristic ensures that the output of the chosen classifier is close to the most contiguous of all candidates for the longest common subsequence. The heuristic may take into account number of match segments, number of gaps between segments, and variance of segment lengths. Losses due to greediness are stemmed by ensuring the heuristic is only followed if the mean of the length of currently matched segments is >n, thus reducing losses post-cleaning.

Because the heuristic is used as an additional measure on top of the match score in dynamic programming, usage of the heuristic can be controlled at each step to vary the contiguity of the final common subsequence outputted. In viewing the contextual feature extraction process as a whole, the contiguity heuristic, along with the cleaning process and variable application of greediness, ensures that sensible text features are extracted while only increasing the time and space complexities by a constant factor.

FIG. 10 is an exemplary set of executable instructions for calculating the contiguity of a string. FIG. 11 is an exemplary set of executable instructions for calculating the mean length of a string, and FIG. 12 is an exemplary set of executable instructions for calculating a longest contiguous subsequence for a pair of strings, such as String A and String B.

FIG. 13 is an exemplary set of executable instructions for creating a feature set from suggested sequences generated by the instructions of FIG. 12. Using a set of user-specified contextual character sequences, a function is created by the system 100 that maps contexts of positive/negative entities into feature vectors to train a classifier model, and similarly to map new contextual instances into a feature vector to input in the trained model.

Advantageously, features are sensitive to the similarity between specified sequences and new contexts, and the ordering of characters within each specified sequence. Additionally, the magnitude of the distance between feature vectors generated from contexts containing OCR errors and feature vectors generated from an error-free context are preferably as small as possible. The proposed feature set generation method is a realignment/windowing method, specified by executing the instructions of FIG. 13.

FIG. 14 illustrates a method having steps that are executed in accordance with the present disclosure. The method includes a step 1405 of receiving annotated documents that comprise annotated fields. Again, this step may include receiving documents annotated by a document annotator. These documents may include documents that have been pre-processed by a base annotator to extract general field types.

Next, the method may include a step 1410 of analyzing the annotated documents to determine contextual information for each of the annotated fields, as well as a step 1415 of determining discriminative sequences using the contextual information. In some instances the method may include a step 1420 of generating a proposed rule of a feature set using the discriminative sequences and annotated fields and a step 1425 of providing the proposed rule or the feature set to a document annotator. The proposed rule may be provided to a rule-based extractor generator, whereas the feature set is provided to a feature-based extractor generator. The rule-based extractor generator is configured to create an extractor rule from the proposed rule using, for example, user feedback. The feature-based extractor generator generates a feature-based vector from the feature set using any of the method described in greater detail above.

FIG. 15 illustrates another exemplary method that includes a specific method for determining discriminative sequences, for example, a more detailed sub-method for step 1415 of FIG. 14. The method may include a step 1505 of determining various longest contiguous common subsequences between aligned pairs of strings of annotated documents. Next, the method includes the step 1510 of determining a frequency of occurrence of similar longest contiguous common subsequences. In some instances, the method allows for selection of the highest ranked similar subsequences. According to some embodiments, the method comprises the step 1515 of generating a proposed rule of a feature set from longest contiguous common subsequences having a desired frequency of occurrence. Again, the feature set may be generated from the subsequences having a high rate of frequency within the annotated documents.

As mentioned previously, the extractor system may use the features to train a classifier module that is used to classify target documents.

FIG. 16 is a flowchart of an exemplary method for annotating documents using a context and contiguity analysis module as described in greater detail above with respect to the context and contiguity analysis module 115 of FIG. 1, hereinafter referred to as the context module 115 for brevity. In general, context module 115 may include a processor and a memory for storing executable instructions (e.g., logic) that cause the system to determine contextual information from training documents and apply that knowledge to other documents in order to annotate the documents in manner that is consistent with the annotations of the training documents. In general, the context module 115 may include any of the features of the exemplary computing system of FIG. 19. The context module 115 includes a memory that stores instructions that when executed by the processor cause the context module 115 to perform the various methods outlined below.

Generally, the method of FIG. 16 is a method that can be utilized to annotate documents using a training corpus that is supplied by an end user. The training corpus includes one or more training documents that have been annotated. For example, the training documents may include any type of textual document such as a contract or agreement that has been annotated to specify words or phrases that have certain attributes such as a contract start date, a contracting party, a term, a price, or other document object. The training corpus may include a single annotated document or a plurality of training documents. In some embodiments, the training documents are of similar structure and type to those that the end user wishes to annotate. That is, a document that the end user desires to automatically annotate using the present technology may be of a similar domain to the documents used to train the system. For example, the context module 115 may automatically annotate a sales contract using an exemplary set of training documents that include other sales contracts for the same parties or the same type of transaction.

Therefore, in some embodiments, the method includes receiving 1605 training documents. As mentioned above, the training documents include annotated words. For example, the training documents may include sales contracts where the effective date, parties, pricing, milestones, and other document features have been annotated.

In some embodiments the context module 115 is configured to determine contextual information surrounding each of the annotated words in the training documents. Using contextual information, the context module 115 can look for similar patterns (e.g., discriminative sequences) in an unannotated document and locate similar document features in the unannotated document.

More specifically, the method may include identifying 1610 a predetermined number of characters preceding and following each annotated word for each of the training documents to determine a context for each of the annotated words. By way of example, the context module 115 may obtain 50 or 100 characters both preceding an annotated word and following the annotated word. For similar annotations, the context module 115 may compare the preceding and following characters for multiple annotations to one another to discover or determine discriminative sequences therein. For example, the context module 115 may determine that the words “begin” and “beginning” always precede a start date annotation. As an aside, the word begin would be a longest common subsequence between the words “begin” and “beginning”. Thus, the system can infer that a date field that is proximate the words “begin” or “beginning” are likely to be start date fields.

As an illustration, FIG. 2 includes a contractual document, which for purposes of explanation may be considered a training document. The contract includes a start date annotation, such as annotation 205. The context module 115 may obtain 100 characters before and after the annotation 205. When examined further, the word “begins” appears roughly within 20 characters of the annotated date field. Other similar contextual features may be determined as well. It is noteworthy that similar words or features in a document may often be preceded or followed by similar words or phrases. These words or phrases provide context or “clues” that the words or features are similar to one another and should be annotated similarly to one another.

According to some embodiments, the method includes processing each annotation of the training documents to look for discriminative sequences in the training documents. Once the context module 115 builds a knowledge base from the training documents, the context module 115 can automatically annotate an unannotated document, which is hereinafter referred to as a target document. More specifically, the context module 115 may, in some instances, generate an extractor rule for each annotated word in a training document. The extractor rule includes not only the annotated word but also its context, which includes the words or characters that preceded or follow the annotated word.

An extractor rule may also be generated when it is determined that a plurality of similar annotated fields (e.g., such as date fields) include similar or substantially similar characters that precede the date field. Thus, the context module 115 may create an extractor rule when two or more training documents include the same or similar contextual information around the same type of annotated field. In other instances, the context module 115 may utilize the annotated fields and context information for a single training document as the basis for creating extract rules.

Prior to processing a target document and determining automatic annotations, the context module 115 may evaluate the target document to determine various features in the target document such as date fields, text fields, monetary fields, tables, lists, or other high level document fields. These features can be assigned (or confirm) a specific type of annotation by the context module 115 according to further aspects of the method.

Generally, the context module 115 applies the extractor rules created from the training document(s). More specifically, the application of the extractor rules includes performing an alignment between extractor rules created from the training documents with text in the target document. This process includes aligning 1615 of an annotated word and its context with characters in the target document. Advantageously, the method includes aligning each possible annotated word in the training documents, and their associated contexts, with characters in the target document.

The method also includes identifying 1620 common sequences between the target document and training corpus, based upon the alignment. For example, the context module 115 may determine that the target document includes the word “begin” within 100 characters preceding a date field. Thus, the context module 115 can infer that the date field is in fact a start date in a contract.

The method includes assigning 1625 an annotation to a word in the target document when a common sequence between the target document and training corpus is found. That is, the annotation comprises an annotation of an annotated word of the training corpus (e.g., annotated training documents) included in the common sequence.

The method is executed for each annotatable word in the target document until the context module 115 has annotated each unannotated word. In some embodiments, the context module 115 may not examine the document for fields that can be annotated and merely looks for discriminative sequence alignments between the training documents and the target document. When a match is found, the context module 115 can then examine the sections of the target document that match the training documents to locate a potential word or document feature, such as a date that can be annotated with an annotation that corresponds to the annotation of the training documents.

In some embodiments, the context module 115 utilizes an exact matching requirement where an alignment between training document sequences and target document sequences must perfectly align before an annotation can be assigned to a word in the target document. In other embodiments, the context module 115 may utilize a fuzzy matching algorithm, such as the windowing algorithm illustrated in FIGS. 9A and 9B to find matching characters between the sequences of the training documents and the target document. The fuzzy match allows for a match between, for example, the words “begin date” as found in three training documents and the words “begin date” found in the target document. Such a sequence may be found in a target document when an optical character recognition (OCR) error is introduced into a target document. Thus, the context module 115 could infer a match between ‘beg’, ‘n’, and ‘date’ when sequences of the training and target documents are aligned.

In some instances, the method includes storing 1630 an annotated word and its context, which includes the characters preceding and following the annotated word, as an extractor rule. Again, an extractor rule may be stored only if a sufficient number of training documents includes the same annotated word and context. For example, if at least three training documents include the same context (e.g., discriminative sequences of characters) as one another, the context module 115 may store the annotated word and context as an extractor rule. In other embodiments, annotated fields and their contexts are stored as extractor rules.

FIG. 17 is a flowchart of another exemplary method for automatically annotating documents using a context module 115. The method includes receiving 1705 training documents. As with the method of FIG. 16, the training documents include annotated words. Next, the method includes identifying 1710 a predetermined number of characters preceding and following each annotated word for each of the training documents to determine a context for each of the annotated words.

Using the annotated words and their contexts, the method includes generating a plurality of feature vectors, where each feature vector includes an annotated field and its context (e.g., characters before and after the annotated word). Once the extractor rules have been created from the training documents, the method includes obtaining a training document from the end user. Again, the target document includes a document that is similar in content and/or formatting to the training documents except that the target document does not include annotated fields.

The method includes dividing 1715 the target document into n-gram strings. In contrast with methods that require an exact match between the extractor rules and the method of FIG. 17 allows for fuzzy or near matching between the feature vectors generated from the training documents and the target document. As mentioned above, this process compensates for OCR errors or other document errors that would preclude detection of a match between the training documents and the target documents. Stated otherwise, but for the document errors, the system would have detected a match between the training documents and the target document for a particular annotation or annotations.

In some embodiments, the method includes identifying 1720 overlapping sequences of characters between the n-gram strings of the target document and the training documents. For example, an n-gram match between, for example, the words “begin date” as found in three training documents and the words “begin date” found in the target document may be found. Such a sequence may be found in a target document when an optical character recognition (OCR) error is introduced into a target document. Thus, the context module 115 would infer a match between ‘beg’, ‘n’, and ‘date’ when sequences of the training and target documents are aligned. The context module 115 would ignore the “1” which is present in the target document. This substantial alignment between the begin date” and “begin date” would be sufficient to justify asserting that a match exists therebetween.

Finally, the method includes assigning 1725 an annotation to each word in the target document when the overlapping sequences between the n-gram strings of the target document and the training documents meet or exceed a threshold. For example, the context module 115 may specify that at least 90% of a feature-based vector (e.g., n-gram model) should correspond to the n-gram text segments of the target document to justify assigning an annotation to a particular text field in the target document. The method may optionally include storing 1730 an identified overlapping sequence as a feature-based vector.

FIG. 18 illustrates another exemplary method for annotating documents using the context module 115. The method includes receiving 1805 training documents that have annotated words. Also, the method includes, for each annotated word, determining 1810 a distance between an annotated word and one or more document features. That is, each annotated word in a training document is measured in terms of a distance between various document features in the training document. Exemplary document features include, but are not limited to a title, header, text block, list, image, link, page number, a footnote, a section identifier, and a document landmark that identifies a discrete section of portion of a document that would be known to one of ordinary skill in the art. A document feature may be referred to generally as a document landmark. It will be understood that a document feature could include any object that can be used as a reference for measuring a distance. For example, an annotated word, such as a start date for a contract may be 3.5 inches from the title. When a plurality of training documents are analyzed, it may be determined that contract start date annotated fields are located typically between 3.1 and 3.8 inches from the title of the document. The context module 115 may generate a rule from this knowledge, such that any date field that falls within the range of 3.1 and 3.8 inches from the title of a document can be presumptively annotated as a contract start date. The rule may also be expressed as a percentage variation from an average rather than a range of distances. For example, the system may calculate an average of 3.4 inches from the title. Also, the context module 115 may create an acceptable boundary range for the average value, such as any value that is +/−15% of the average calculated distance.

In other embodiments, the method may include calculating 1815 an averaged location for each annotated word, where the averaged location defines an averaged placement of an annotated word within a training document. Thus, the context module 115 would determine a document length and measure the placement of the annotated word from begging and the end of the document, or the beginning and end of a page of the document, or any combinations or variations thereof. This averaged location may include an average of all similar annotations in a plurality of documents. For example, a contract end date annotation is found to be an average of five inches from the beginning of training documents and an average of nine inches from the bottom of the first page of each of the training documents.

In other embodiments, the distance value calculated for an annotation may include not only a relative location of the word in the document, but also a distance between the word and other document features as described above. This value may be expressed as a single numerical value. In some instances, the context module 115 may weight each of the distance values so as to attribute more value or importance to a particular distance value. For example, distance from the title (Dt) may be weighted more highly than distance from nearest date value (Dv), but each of these distance values forms a part of the calculation. Ostensibly, the higher the calculated value, the more likely it is that the annotation of the training documents and the potential annotation of the target document are a match. A rudimentary exemplary calculation would include (Dt*0.7+Dv*0.3)/2. The coefficients assigned to the individual distance values can be assigned by the context module 115 and represent the relative importance of each distance value in the overall calculation.

Once the training documents have been processed, the method includes for each word of a target document, determining 1820 a distance between each word and one or more document features. In some instances, the document features include the same or similar document features as those present in the training documents. Also, the method includes calculating 1825 an averaged location for each word. Again, the averaged location defines an averaged placement of an annotated word within a target document.

Next, the method includes identifying 1830 matches between the distance and averaged location of words of the target document and the distances and averaged location of at least one of the annotated words of the training documents. When a match is found, the method includes assigning 1835 an annotation to a word of the target document that matches the at least one of the annotated words of the training documents.

FIGS. 19A-C illustrates various processes for determining an averaged location of an annotatable (or annotated) word in a document. FIG. 19A illustrates a plurality of documents Doc 1, Doc 2, and Doc 3. Each of these documents includes an annotated field such as Jan. 1, 2013 in Doc 1. The process of calculating an averaged location for each of the annotated dates in the documents relative to a beginning of a section or the beginning of the document itself. In some instances, the process involves using the same feature for each document. That is, the process involves comparing the location of each similar annotation to the same reference point, such as a section beginning. The averaged distance is the average of a sum of these determined distances.

FIG. 19B also illustrates an averaged location determination process where the averaged locations of annotations are determined from distances between annotations and their preceding sections of a document. In this example, the distance that is determined is a distance between an annotated date field and a page number. FIG. 19C also illustrates an averaged location determination process where the averaged locations of annotations are determined relative to different sections in the same document. For example, a first annotated date of Jan. 1, 2013 may precede a section title of “5” and a second annotated date of May 1, 2013 may precede a section title of “7” that is located further into the document than section “5”.

In one embodiment, the present technology provides a method for not only determining context in a document but also for extracting meaning relative to statements in a document. For example, the present technology can be used to extract from a document meaning surrounding a particular topic or keyword. An example topic would be sub-contracting. In this example, a contractual document can be processed to determine if there are provisions relating to sub-contracting by a contractor or party to an agreement. Sentences that mention sub-contracting can be analyzed to determine meaning so as to infer if sub-contracting is allowed, not allowed, or conditionally allowed. These possibilities are referred to as categories or outcomes.

The present technology can leverage training documents that comprise words which were annotated with categories of meaning. Again, the annotations and meaning can relate to the position of words in the document relative to other words or features, the context of words/characters that precede or succeeded an annotated word, as well as other features in a document that would be indicative of meaning for a particular word or topic (e.g., subject matter or domain) associated with that word.

The present technology can initially review a target document for words or phrases (or other sets of characters) that are associated with sub-contracting. The training corpus is used to automatically classify words by determining the meaning of the located words. Again, this includes, in one embodiment, comparing the context of the word to a context of an annotated word one or more training documents. If a match is found there between, the word in the target document can be assigned a classification or category. For example, if the word sub-contractor is found in a target document, the context for that word can be determined by looking at the word in the context of the document. The present technology can evaluate the sentence in which the word belongs, as a whole and determine meaning that can be attributed to the word. Thus, by reviewing the sentence as a whole, or other words in the sentence, it can be determined if sub-contracting is allowed, not allowed, or conditionally allowed. By way of example, consider the sentence “Subject to the provision of Section 6.2, sub-contracts are permissible under this agreement.” The present technology locates the presence of the word “sub-contracts” and then locates other words within the sentence such as “permissible” and “subject to”, via comparison to a training corpus. From consulting a training corpus, the present technology extracts meaning for this sentence and classifies this sentence and the document as providing conditional provisions for sub-contracting.

FIG. 20 illustrates an exemplary computing device 2000 that may be used to implement an embodiment of the present systems and methods. The system 2000 of FIG. 20 may be implemented in the contexts of the likes of computing devices, networks, servers, or combinations thereof. The computing device 2000 of FIG. 20 includes a processor 2010 and main memory 2020. Main memory 2020 stores, in part, instructions and data for execution by processor 2010. Main memory 2020 may store the executable code when in operation. The system 2000 of FIG. 20 further includes a mass storage device 2030, portable storage device 2040, output devices 2050, user input devices 2060, a display system 2070, and peripherals 2080.

The components shown in FIG. 20 are depicted as being connected via a single bus 2090. The components may be connected through one or more data transport means. Processor 2010 and main memory 2020 may be connected via a local microprocessor bus, and the mass storage device 2030, peripherals 2080, portable storage device 2040, and display system 2070 may be connected via one or more input/output (I/O) buses.

Mass storage device 2030, which may be implemented with a magnetic disk drive or an optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor 2010. Mass storage device 2030 can store the system software for implementing embodiments of the present technology for purposes of loading that software into main memory 2020.

Portable storage device 2040 operates in conjunction with a portable non-volatile storage medium, such as a floppy disk, compact disk or digital video disc, to input and output data and code to and from the computing system 2000 of FIG. 20. The system software for implementing embodiments of the present technology may be stored on such a portable medium and input to the computing system 2000 via the portable storage device 2040.

Input devices 2060 provide a portion of a user interface. Input devices 2060 may include an alphanumeric keypad, such as a keyboard, for inputting alphanumeric and other information, or a pointing device, such as a mouse, a trackball, stylus, or cursor direction keys. Additionally, the system 2000 as shown in FIG. 20 includes output devices 2050. Suitable output devices include speakers, printers, network interfaces, and monitors.

Display system 2070 may include a liquid crystal display (LCD) or other suitable display device. Display system 2070 receives textual and graphical information, and processes the information for output to the display device.

Peripherals 2080 may include any type of computer support device to add additional functionality to the computing system. Peripherals 2080 may include a modem or a router.

The components contained in the computing system 2000 of FIG. 20 are those typically found in computing systems that may be suitable for use with embodiments of the present technology and are intended to represent a broad category of such computer components that are well known in the art. Thus, the computing system 2000 can be a personal computer, hand held computing system, telephone, mobile computing system, workstation, server, minicomputer, mainframe computer, or any other computing system. The computer can also include different bus configurations, networked platforms, multi-processor platforms, etc. Various operating systems can be used including UNIX, Linux, Windows, Macintosh OS, iOS, and other suitable operating systems.

Some of the above-described functions may be composed of instructions that are stored on storage media (e.g., computer-readable medium). The instructions may be retrieved and executed by the processor. Some examples of storage media are memory devices, tapes, disks, and the like. The instructions are operational when executed by the processor to direct the processor to operate in accord with the technology. Those skilled in the art are familiar with instructions, processor(s), and storage media.

It is noteworthy that any hardware platform suitable for performing the processing described herein is suitable for use with the technology. The terms “computer-readable storage medium” and “computer-readable storage media” as used herein refer to any medium or media that participate in providing instructions to a CPU for execution. Such media can take many forms, including, but not limited to, non-volatile media, volatile media and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as a fixed disk. Volatile media include dynamic memory, such as system RAM. Transmission media include coaxial cables, copper wire and fiber optics, among others, including the wires that comprise one embodiment of a bus. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM disk, digital video disk (DVD), any other optical medium, any other physical medium with patterns of marks or holes, a RAM, a PROM, an EPROM, an EEPROM, a FLASHEPROM, any other memory chip or data exchange adapter, a carrier wave, or any other medium from which a computer can read.

Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a CPU for execution. A bus carries the data to system RAM, from which a CPU retrieves and executes the instructions. The instructions received by system RAM can optionally be stored on a fixed disk either before or after execution by a CPU.

Computer program code for carrying out operations for aspects of the present technology may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present technology has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. Exemplary embodiments were chosen and described in order to best explain the principles of the present technology and its practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Aspects of the present technology are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present technology. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the technology to the particular forms set forth herein. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments. It should be understood that the above description is illustrative and not restrictive. To the contrary, the present descriptions are intended to cover such alternatives, modifications, and equivalents as may be included within the spirit and scope of the technology as defined by the appended claims and otherwise appreciated by one of ordinary skill in the art. The scope of the technology should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the appended claims along with their full scope of equivalents.

	Number	Date	Country
Parent	13948069	Jul 2013	US
Child	14452484		US

	Number	Date	Country
Parent	14452484	Aug 2014	US
Child	14543821		US

INFORMATION EXTRACTION AND ANNOTATION SYSTEMS AND METHODS FOR DOCUMENTS

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

US Classifications

International Classifications

Abstract

Description

Claims

CROSS REFERENCE TO RELATED APPLICATIONS

Provisional Applications (1)

Continuations (1)

Continuation in Parts (1)