Automatically identifying multi-word expressions

Abstract

A facility for identifying multi-word expressions in a natural language sentence is described. The facility provides the sentence to each of multiple natural language processing modules including a first module, a second module, and a third module. Each natural language processing module uses a different approach to identify a multi-word expression and a type of the multi-word expression. Upon determining that the multiple identifiers of the multi-word expression differ, the facility determines the multi-word expression using a resolution process, which can involve a logical rule set or a machine learning model.

Description

BACKGROUND

Various automatic natural language processing and natural language understanding applications process natural language text based upon recognizing noun phrases and verb phrases.

Some noun phrases consist of only a single word, while others are each made up of multiple words. Similarly, some verb phrases are made up of a single word, while others contain multiple words. Multiple-word verb phrases and multiple-word noun phrases are referred to collectively herein as “multi-word expressions” or “MWEs.”

Table 1 below contains a number of sample sentences in which noun MWEs are identified using bold formatting, and verb MWEs are identified by italic formatting. These sample sentences are typical of those taken from authority documents processed for compliance purposes.

TABLE 1
1. a system and information
   integrity                                 policy              .
2. personal data.
3. compliance                                 monitoring                                 statistics to the Board of
   Directors               and other critical stakeholders, as necessary.
4. out of scope systems from in scope systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the facility operates.

FIG. 2 is a flow diagram showing a process performed by the facility in some embodiments to automatically identify MWEs that occur in input text.

FIG. 3 is a data flow diagram showing aspects of the facility's performance of the process shown in FIG. 2.

FIG. 4 is a dependency diagram showing dependencies discerned between tokens by the facility in a sample sentence.

FIG. 5 is a flow diagram showing a process performed by the facility in each constituent model to identify MWEs occurring in the pre-processed input text.

FIG. 6 is a text analysis diagram showing a version of the input text that the facility has processed using the NLP pipeline to attach parts of speech to each token.

FIG. 7 is a relationship diagram showing ways in which the facility can expand the noun term director based upon relationships in the dictionary.

FIG. 8 is a flow diagram showing a process performed by the facility in some embodiments to build a machine learning model to function as its constituent model result evaluation module.

FIG. 9 is a model diagram showing the facility's initial training and application of the machine learning model.

FIG. 10 is a flow diagram showing a process performed by the facility in some embodiments to perform supplemental training for the machine learning model.

FIG. 11 is a display diagram showing a sample display presented by the facility in some embodiments to obtain input from a human editor about whether to remove an MWE from the dictionary.

FIG. 12 is a model diagram showing the facility's supplemental training of the machine learning model.

DETAILED DESCRIPTION

The inventors have recognized that, while conventional dictionaries are reasonably complete in identifying single-word nouns and verbs that can be matched to sentences being analyzed, they include only a very small percentage of MWEs. Additionally, they have recognized that the particular MWEs used varies dramatically across different domains of writing.

In response to recognizing the above disadvantages of using conventional dictionaries as a basis for recognizing noun phrases and verb phrases that include MWEs, the inventors have conceived and reduced to practice a software and/or hardware facility that automatically identifies MWEs in input sentences (“the facility”).

In particular, the facility uses two or more constituent models that each employ a different approach to generate analysis results identifying MWEs that occur in an input sentence. The facility then uses constituent model result evaluation module to derive overall MWE identification results from the results produced by the constituent models. In some embodiments, the constituent model result evaluation module is a logical ruleset applied by the facility. In some embodiments, the constituent model result evaluation module is a machine learning model—such as a reinforcement learning model—constructed, trained, and applied by the facility.

In some embodiments, one of the facility's constituent models is a transformer, such as a Bidirectional Encoder Representation from Transformers (“BERT”) transformer. In various embodiments, the facility uses the results of empirical testing to (1) select one among multiple transformer types, transformer training sets, and/or transformer configurations for exclusive use by the constituent model, or to (2) weight results produced by two or more different transformer types, transformer training sets, and/or transformer configurations to obtain a weighted aggregation for output by the transformer constituent model.

In some embodiments, one or more of the facility's constituent models implement linguistic analysis algorithms, such as noun chunking algorithms and/or verb chunking algorithms. In various embodiments, the facility uses the results of empirical testing to (1) select one among multiple linguistic analysis algorithms for exclusive use by this constituent model, or to (2) weight results produced by two or more different linguistic analysis algorithms to obtain a weighted aggregation for output by the constituent model.

In some embodiments, one of the facility's constituent models accesses a dynamic dictionary to identify in the input text MWEs that have entries in the dictionary. In some embodiments, the dictionary's MWE entries have been added to it in response to input from a linguist and/or a subject matter expert certifying that they are proper MWEs. In some embodiments, this dictionary matches the longest MWEs contained by both the input text and the dictionary.

In some embodiments, the facility uses a constituent model result evaluation module to arbitrate among different results outputted by the different constituent models recognizing MWEs occurring in particular input text, as described in greater detail below. In some embodiments, the constituent model result evaluation module also specifies whether particular MWEs identified by certain constituent models should be automatically added to the dictionary, or should be queued for review by a linguist and/or subject matter expert for discretionary addition to the dictionary. This latter determination is often referred to herein as “nomination” of the MWEs recommended for addition to the dictionary.

In some embodiments, the facility is used to identify MWEs as part of natural language analysis performed by a compliance tool. Compliance tools facilitate an organization's adherence to rules of various kinds that govern their business, and assessing (“auditing”) that adherence. These rules are expressed in authority documents, which can include, for example: statutes, regulations, regulatory directives or guidance, contractual obligations, standards, auditing guidelines, safe harbors, best practice guidelines, vendor documentation, and procedures established by the organization for its own operation. In some cases, a compliance process involves some or all of the following phases: selecting and obtaining copies of a group of authority documents that applies to the organization; identifying the expressions of rules (“citations”) that occur in the authority documents; performing natural language understanding analysis of the citations to determine the rules (“mandates”) that they express; deduplicating the mandates across the group of authority documents—and within individual authority documents—to obtain “controls” (or “common controls”) that each represent a set of mandates that are equivalent, and are each linked to that set of mandates; constructing an audit questionnaire from the controls that efficiently covers compliance with all of the authority documents in the group; and using the established structure of citations, mandates, controls, and audit questions and answers to establish that the answers to audit questions demonstrate compliance with the authority documents in the group. In some cases, documents, citations, mandates, and/or controls are constructed with reference to data objects called “terms” that constitute dictionary entries for words or phrases occurring in those higher-level data objects.

By performing in some or all of the ways described above, the facility identifies a much more complete list of noun phrases and/or verb phrases quickly, using limited computing resources, enabling higher-quality natural language processing results.

Also, the facility improves the functioning of computer or other hardware, such as by reducing the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be performed by less capable, capacious, and/or expensive hardware devices, and/or be performed with less latency, and/or preserving more of the conserved resources for use in performing other tasks or additional instances of the same task. For example, the facility conserves processing and user interface resources that would have been applied to facilitating a person's manual analysis of entire documents to identify the contained MWEs. As a result, cheaper, less powerful devices can be substituted to achieve the same level of performance, or the same devices can be used with excess processing capacity remaining for performing additional desirable tasks.

FIG. 1 is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the facility operates. In various embodiments, these computer systems and other devices 100 can include server computer systems, cloud computing platforms or virtual machines in other configurations, desktop computer systems, laptop computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, etc. In various embodiments, the computer systems and devices include zero or more of each of the following: a processor 101 for executing computer programs and/or training or applying machine learning models, such as a CPU, GPU, TPU, NNP, FPGA, or ASIC; a computer memory 102 for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel, and device drivers; a persistent storage device 103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive 104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer-readable medium; and a network connection 105 for connecting the computer system to other computer systems to send and/or receive data, such as via the Internet or another network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like. While computer systems configured as described above are typically used to support the operation of the facility, those skilled in the art will appreciate that the facility may be implemented using devices of various types and configurations, and having various components.

FIG. 2 is a flow diagram showing a process performed by the facility in some embodiments to automatically identify MWEs that occur in input text. In act 201, the facility receives a unit of input text, such as a sentence. In act 202, the facility pre-processes the input text to prepare it for consumption by the constituent models.

FIG. 3 is a data flow diagram showing aspects of the facility's performance of the process shown in FIG. 2. It can be seen that the facility subjects input text 301 to a preprocessor 310 to obtain pre-processed input text 311. Aspects of the preprocessing performed by the facility in some embodiments are described below. In some embodiments, the aspects of the facility's preprocessing are discussed herein as part of an “NLP pipeline” operated by the facility.

In various embodiments, the facility's preprocessing of the input text includes:

• • 1. The facility tokenizes the input text, converting it from original plain text form into word tokens. An example of tokenization is shown below in Table 2 for the third example sentence in Table 1 above.

TABLE 2
Report
compliance
monitoring
statistics
to
the
Board
of
Directors
and
other
critical
stakeholders
,
as
necessary
.

• • 2. In subword tokenization, the facility converts word tokens into subword units to increase the amount of information per token while minimizing vocabulary size. For example, in some embodiments, the facility assigns a different token value to each individual character; then defines word fragment tokens identified by their own token numbers by the token numbers of the characters they contain; then defines a word token having its own token number based upon the token numbers of the word fragment tokens that make it up. An example of this based upon the word “faster” is shown below in Table 3.

TABLE 3
f (6) a (1) s (19) t (20) e (5) r (18) → fast (19731) er (288) → faster
(18277)

• • 3. In performing sentence splitting, the facility divides raw text that may contain multiple sentences into individual sentences.
  • 4. The facility performs part of speech tagging by assigning a part of speech tag to tokens, such as adjective, adverb, conjunction, noun, verb, article, punctuation, etc. An example of part of speech tagging is shown below in Table 4.

TABLE 4
Report (NOUN) compliance (NOUN) monitoring (NOUN) statistics
(NOUN) to (ADP) the (DET) Board (NOUN) of (ADP) directors (NOUN)
and (CONJ) other (ADJ) critical (ADJ) stakeholders (NOUN) , (PUNCT)
as (ADP) necessary (ADJ) . (PUNCT)

• • 5. The facility performs lemmatization by converting each word token into its lemma, i.e., its base dictionary form. For example, the facility lemmatizes the word “stakeholders” in the example sentence to its singular form “stakeholder”.
  • 6. The facility performs named entity recognition by detecting entities—such as countries, organization, people, etc.—numbers, dates, and times.
  • 7. The facility performs dependency analysis by parsing the text and detecting linguistic relationships between the tokens.

FIG. 4 is a dependency diagram showing dependencies discerned between tokens by the facility in a sample sentence. In particular, each directed arc among tokens 401-417 shows a dependency of one token on another. For example, the arc from token 403 to token 402 reflects a dependency of the token “monitoring” on the token “compliance.”

Returning to FIG. 2, in act 203, the facility initiates each constituent model against the pre-processed input text obtained in act 202.

Returning to FIG. 3, it can be seen that the following constituent models are initiated by the facility against pre-processed input text 311: a transformer constituent model 321, a noun chunk constituent model 331, a verb chunk constituent model 341, and a dictionary constituent model 351. These constituent models each generate their own result, results 322, 332, 342, and 352, respectively.

FIG. 5 is a flow diagram showing a process performed by the facility in each constituent model to identify MWEs occurring in the pre-processed input text. In act 501, the facility receives the pre-processed input text. In act 502, the facility analyzes the pre-processed input text received in act 501 to produce a result that identifies MWEs occurring in the input text. In act 503, the facility outputs the obtained result as the constituent model result. After act 503, this process concludes.

Those skilled in the art will appreciate that the acts shown in FIG. 5 and in each of the other flow diagrams discussed herein may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.

The operation of the facility's constituent models is discussed in greater detail below.

In some embodiments, for the transformer constituent model, the facility uses a BERT-Base multilingual case model which has been trained on 104 languages with 12-layer, 768-hidden, 12-heads, and 110 M parameters. Additional details about configuring and using BERT are available in Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018, Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, which is herein incorporated by reference in its entirety. In some embodiments, the facility uses an ALBERT-xxlarge-v1 model with 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, and 223 M parameters. Additional details about configuring and using ALBERT are provided by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, 2019, Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909.11942, which is hereby incorporated by reference in entirety. In some embodiments, the BERT model, ALBERT model, or both are trained using a DiMSUM data set. Details about this data set are provided by DiMSUM 2016 Shared Task Data, available at github.com/dimsuml6/dimsum-data, which is hereby incorporated by reference in its entirety.

In various embodiments, the facility uses a noun chunk constituent model that uses some or all of the following bases for identifying noun MWEs: NC head count—the nouns in a sentence where the nouns are the root of the sentence; NC all count—all of the tokens with a POS tag NOUN; NC spacy count—all the noun chunks identified by a built-on noun chunk iterater in a Spacy tool; NC textacy count—Spacy noun chunks omitting determiners such as “a” or “the”; and/or NC noadj count—the Spacy noun chunks omitting determiners and adjectives. Details about the Spacy tool are provided by TextCategorizer, available at spacy.io/api/textcategorizer, which is hereby incorporated by reference in its entirety.

In some embodiments, the facility uses a verb chunk constituent model that uses some or all of the following bases for identifying verb MWEs: VC head count—the verbs in a sentence where the verbs are the root of the sentence; VC all count—all the tokens with a POS tag VERB; VC consecutive count— consecutive tokens with a POS tab VERB; VC immhead count—tokens that have dependency labels that are among XCOMP, ACOMP, AUXPASS, AUX, PRT, NEG, CONJ, ROOT and the corresponding POS tag of the head tokens are among VERB, NOUN, or PRONOUN; and/or VC head left right count—head verbs whose children have dependency labels among XCOMP, ACOMP, AUXPASS, AUX, PRT, NEG, CONJ, ROOT—construct the entire phrase from leftmost child to rightmost child.

In some embodiments, the dictionary constituent model acts on a version of the input text that is pre-processed in the NLP pipeline to include parts of speech. FIG. 6 is a text analysis diagram showing a version of the input text that the facility has processed using the NLP pipeline to attach parts of speech to each token. It can be seen that, for each of tokens 601-617, the facility has attached the part of speech. For example, the facility has attached the part of speech NOUN to the token “report” 601, and the part of speech CONJ to the token “and” 610.

In some embodiments, the dictionary constituent model processes each token identified by the NLP pipeline as a verb by looking that token up in the dictionary and verifying that the dictionary contains an entry for that word having the same part of speech, i.e., verb. The facility then performs word expansion by looking up associated broader terms that include the base verb and words that are adjacent to the left or right in the pre-processed input text until these broadened verbs no longer match input terms or overlap MWEs previously detected in the input text. The facility uses the broadest match in the dictionary. If the word or part of speech does not exist in the dictionary, then the facility skips detecting verb MWEs.

In some embodiments, for tokens that the NLP pipeline identifies as nouns, the facility looks each up in the dictionary and verifies that it contains an entry for that token having the same part of speech, i.e., noun. The facility performs word expansion by looking at the associated broader terms that include the base noun and words adjacent to the left or right until the broadened nouns no longer match the input terms, or overlap MWEs previously detected in the input text. The facility uses the broadest match in the dictionary. If the word or part of speech does exist in the dictionary, then the facility skips detecting noun MWEs.

FIG. 7 is a relationship diagram showing ways in which the facility can expand the noun term “director” based upon relationships in the dictionary. A central node 700 for the noun “director” is related or synonymous with a number of other noun terms in nodes 711-715; it is the linguistic child of noun terms in nodes 721-738; and it is the linguistic parent of noun terms in nodes 741-769.

Returning to FIG. 2, in act 204, the facility collects the output generated by each constituent model, that is, the noun MWEs identified by each constituent model and the verb MWEs identified by each constituent model. Table 5 below identifies, in its second through fifth rows, the result reported by each of the four constituent model models, along with a comparison of that result to a result generated by a human expert shown in the first row of the table as the “gold standard.”

TABLE 5
Approach	IOB tagged sentence with MWE type	Score
UCF-GRC	Report compliance   monitoring   statistics to the	Accurate
(Gold standard)	Board   of   Directors and other critical stakeholders,
	as necessary.
ALBERTxxlarge-	Report compliance   monitoring   statistics to the	Inaccurate
v1	Board of Directors and other critical stakeholders,
	as necessary.
nc textacy count	Report   compliance   monitoring   statistics to the	Inaccurate
	Board of Directors and other critical stakeholders,
	as necessary.
vc head count	compliance monitoring statistics to the	Accurate for
	Board of Directors and other critical	verbs only
	stakeholders, as necessary.
UCF Dictionary	compliance   monitoring   statistics to the	Inaccurate
	Board   of   Directors and other critical stakeholders,
	as necessary.
UCF (Combined	Report compliance   monitoring   statistics to the	Accurate
approach,	Board   of   Directors and other critical stakeholders,
detection	as necessary.
techniques)

The sixth row of the table contains the ultimate result determined for this input text by the facility, as described below.

In act 205, the facility applies the constituent model result evaluation module to the output of the constituent models collected in act 204. This constituent model result evaluation module 360 is shown FIG. 3.

In some embodiments, the constituent model result evaluation module applied by the facility is the following ruleset:

• • 1. When all three MWE detection methods return the same set of lexical tokens, the facility marks those lexical tokens as MWEs, and suggests no new MWEs to add to the dictionary.
  • 2. When all three MWE detection methods return different sets of lexical tokens, then the dictionary wins—i.e., the result of the dictionary constituent model is adopted as the overall result. If a linguistic algorithm override rule exists to properly classify a commonly misclassified MWE type, then the facility marks MWEs detected by the transformer architecture and linguistic algorithms as MWEs and suggests them as additions to the dictionary. Where such an override does not apply, the facility marks the MWEs identified using the dictionary, and does not suggest any new MWEs.
  • 3. When all three MWE detection methods return different sets of lexical tokens, then the dictionary wins, and the facility marks the MWEs identified by the dictionary, and suggests the MWEs detected by the transformer architecture and linguistic algorithms as additions to the dictionary.
  • 4. When the MWEs returned by the Unified Compliance dictionary were a superset of the MWEs detected by the transformer architecture and linguistic algorithms then the facility suggests those MWEs as additions to the dictionary, and the MWE returned by the dictionary won.
  • 5. When the MWEs detected by the transformer architecture and linguistic algorithms were supersets of the MWEs returned by the dictionary then the facility suggests those MWEs as additions to the dictionary, and the MWEs returned by the algorithms won.

In various embodiments, the facility uses various other rulesets in act 205 to arbitrate among the constituent model outputs and suggest MWEs for addition to the dictionary.

In some embodiments, the constituent model result evaluation module applied by the facility in act 205 is a machine learning model. The facility's construction and application of this machine learning model are discussed in greater detail below in connection with FIGS. 8-12.

In act 206, from the application of the constituent model result evaluation module in act 205, the facility determines and outputs an overall result identifying the MWEs in the input text and specifying for each whether it is a noun MWE or a verb MWE. This overall result 361 is shown in FIG. 3. In act 207, from the application of the constituent model result evaluation module in act 205, the facility determines and stores MWE dictionary nominations. These nominated MWEs 362 are shown in FIG. 3. In some embodiments, the facility automatically adds nominated MWEs to the dictionary. In some embodiments, the facility provides a user interface in which human editors, such as linguists and/or domain experts review MWEs to determine whether each should be added to the dictionary.

After act 207, the facility continues in act 201 to receive the next unit of input text.

FIG. 8 is a flow diagram showing a process performed by the facility in some embodiments to build a machine learning model to function as its constituent model result evaluation module. In some embodiments, the machine learning model is a reinforcement machine learning model, such as one implemented using, the Gym reinforcement learning toolkit from OpenAI, described at gym.openai.com. In act 801, the facility uses gold standard data to train the machine learning model.

FIG. 9 is a model diagram showing the facility's initial training and application of the machine learning model. In initial training phase 910, the facility trains the machine learning model 900 data using contents of a gold standard data set 920. In particular, the gold standard data set contains a large number of input text sentences that have been tagged by a human expert to identify the MWEs contained by the sentence. For each sentence, the facility provides a training observation to the model made up of independent variable values independent variable values. First, the facility provides the text 931 of the sentence to the model as an independent variable value. Further, the facility performs NLP preprocessing as described above to obtain NLP pipeline contents 932, which it also provides to the model as an independent variable value. The facility also applies the constituent models to the input text and NLP pipeline contents in order to generate a result for the input text 933 for each constituent model; the facility provides these constituent model results to the model as independent variable values. The facility uses the tagging in the gold standard data set to provide an overall result 934 for the sentence the model as a dependent variable. By comparing the MWEs identified by the overall result each to the dictionary, the facility identifies those MWEs identified by the overall result for which the dictionary does not contain an entry; the facility provides these MWEs as nominated MWEs to the model as a dependent variable.

In some embodiments, training a model involves evaluating the following three reward functions for each of the MWEs identified by each of the constituent models. A first reward function is based on whether the MWE identified by the constituent model matches an MWE identified in the overall result—that is, the tagging of the input sentence in the gold standard dataset. A second reward function is based on whether the MWE identified by the constituent model matches the MWE type—noun or verb—identified for this MWE in the overall result. A third reward function is based on whether the constituent model identified all of the MWEs identified in the overall result. In some embodiments, the facility provides an identifier for each MWE identified in any of the constituent model results, so that in its training the model can associate each set of reward function values with the identity of the MWE that it corresponds to. In some embodiments, these identifiers are word vectors, such as those described in Vectors, available at spacy.io/api/vectors, which is hereby incorporated by reference in its entirety. In some cases, the use of word vectors as MWE identifiers allows the model to assess the level of similarity between pairs of MWEs as a basis for treating similar MWEs similarly.

Through this training process, the facility establishes a state of the machine learning model that enables it to predict dependent variable values based upon independent variable values, as discussed below.

Returning to FIG. 8, after act 801, in act 802, the facility applies the machine learning model trained in act 801 to new instances of input text, such as sentences that occur in a new document, in which MWEs are to be identified. In act 802, the facility is applying the machine learning model in place of the rule set as its constituent model result evaluation module. After act 802, the facility continues in act 802 in order to apply the model to the next input text sentence.

Returning to FIG. 9, in model application phase 960, the facility retrieves input text 981 from a new document 970, and submits it to the model as an independent variable value. The facility performs NLP preprocessing on the input text to obtain NLP pipeline contents 982, which it submits to the model as an independent variable value. The facility applies the constituent models to the input text and NLP pipeline contents to obtain constituent model results 983, which it submits to the model as an independent variable value. Based upon the submitted independent variable values, the model outputs as a dependent variable value an overall result 994 for the input text that identifies the MWEs that occur in the input text, and, for each, whether it is a noun MWE or verb MWE. The model also outputs as a dependent variable value and identification of nominated MWEs, each of which is identified in at least one constituent model result and not in the dictionary.

FIG. 10 is a flow diagram showing a process performed by the facility in some embodiments to perform supplemental training for the machine learning model. In act 901, the facility receives a manual determination identifying a MWE that should be removed from the dictionary. In various embodiments, this manual determination is generated and received in various ways, such as in connection with a systematic or anecdotal review of MWEs in the dictionary, particularly ones that have been added in response to their nomination by the facility; in connection with a systematic or anecdotal review of new input text sentences that have been tagged as including the MWE; etc.

FIG. 11 is a display diagram showing a sample display presented by the facility in some embodiments to obtain input from a human editor about whether to remove an MWE from the dictionary. The display 800 shows a copy 801 of a MWE—such as an MWE that has been added to the dictionary based on its nomination by the facility—as well as a prompt to select either a Yes control 811 to delete the MWE from the dictionary or a No control 812 to omit to delete the MWE from the dictionary. The facility also presents information 820 about the occurrences of the MWE in a corpus of documents to which the dictionary relates. This information includes sample sentences 821 and 831 that occur in documents of the corpus, as well as corresponding frequency counts 822 and 832 indicating the number of times each of the sentences containing the MWE occur in documents of the corpus. In some embodiments, when the user activates the Yes control or the No control, the facility updates the display to show the another MWE.

While FIG. 11 shows a display whose formatting, organization, informational density, etc., is best suited to certain types of display devices, those skilled in the art will appreciate that actual displays presented by the facility may differ from those shown, in that they may be optimized for particular other display devices, or have shown visual elements omitted, visual elements not shown included, visual elements reorganized, reformatted, revisualized, or shown at different levels of magnification, etc.

Returning to FIG. 10, in act 1002, the facility removes the MWE identified in act 1001 from the dictionary. In act 1003, the facility trains the machine learning model to not include the MWE identified in act 1001 in any overall results predicted by the model, and to not nominate this MWE for addition to the dictionary. After act 1003, the facility continues in act 1001 to receive the next manual determination.

FIG. 12 is a model diagram showing the facility's supplemental training of the machine learning model. As part of the supplemental training, based on the manual determination received in act 1001, the facility constructs one or more MWE exclusion observations 1220. In some embodiments, the facility constructs an MWE exclusion observation for each input text sentence in which the MWE identified by the manual determination occurs. In some embodiments, the facility constructs an MWE exclusion observation for each of a proper subset of input text sentences in which the MWE identified by the manual determination occurs. For each MWE exclusion observation, the facility proceeds as follows. The facility provides the observation's input text 1231 to the model as an independent variable value. The facility performs NLP preprocessing to generate NLP pipeline contents 1232 from the input text, which the facility provides to the machine learning model as an independent variable value. The facility applies the constituent models to the input text and NLP pipeline contents to obtain constituent model results, which it provides to the model as independent variable values. The facility provides to the machine learning model as a dependent variable value an overall result 1234 from the MWE exclusion observation from which the facility has removed the identified MWE. Similarly, the facility provides to the machine learning model as a dependent variable value nominated MWEs 1235 from the MWE exclusion observation from which the facility has removed the identified MWE. By submitting these MWE exclusion training observations to the model, the facility enables the model to adjust its state in a way that makes it less likely that the model will include the MWE in the future in its predictions about overall result or nominated MWEs.

The various embodiments described above can be combined to provide further embodiments. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and/or listed in the Application Data Sheet are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

Claims

1. A method in a computing system to establish a trained machine learning model, the method comprising: accessing a plurality of training examples each comprising: a sentence;information identifying one or more multi-word expressions occurring in the sentence; andfor each multi-word expression identified as occurring in the sentence, an indication of whether the multi-word expression is a noun multi-word expression or a verb multi-word expression;for each of the plurality of training examples: for each of a plurality of constituent models, invoking the constituent model against the training example's sentence to obtain a constituent model result for the training example's sentence that identifies one or more portions of the training example's sentence each as a multi-word expression and specifies whether each multi-word expression is a noun multi-word expression or a verb multi-word expression;constructing a training observation corresponding to the training example that comprises: independent variable values comprising: the training example's sentence; andthe constituent model result obtained for each of the plurality of constituent models; anddependent variable values comprising: the training example's information identifying one or more multi-word phrases occurring in the sentence; andusing the constructed training observation to train the machine learning model to predict dependent variable values based on independent variable values.

2. The method of claim 1, further comprising persistently storing the trained machine learning model.

3. The method of claim 1, further comprising, for a distinguished training example, selecting at least one portion of the training example's sentence identified as a multi-word expression in a constituent model result, none of the selected portions having entries in a dictionary,

4. The method of claim 1 wherein the training for each training observation comprises: for each of the plurality of constituent models: for each of the portions of the training example's sentence identified as a multi-word expression by the constituent model result obtained for the constituent model: evaluating a first reward function based on whether the identified portion matches a multi-word expression identified by the training example;evaluating a second reward function based on whether the specification by the constituent model's constituent model result of whether the identified portion is a noun multi-word expression or a verb multi-word expression matches the training example's indication of whether the multi-word expression is a noun multi-word expression or a verb multi-word expression; andevaluating a third reward function based on whether the constituent model result identified all of the multi-word expressions identified by the training example.

5. The method of claim 1, further comprising: receiving a distinguished sentence not among the plurality of training examples;subjecting the distinguished sentence to natural language preprocessing to obtain natural language preprocessing results for the distinguished sentence;for each of the plurality of constituent models, invoking the constituent model against the distinguished sentence and the natural language preprocessing results for the distinguished sentence to obtain a constituent model result for the distinguished sentence that identifies one or more portions of the distinguished sentence each as a multi-word expression and specifies whether each multi-word expression is a noun multi-word expression or a verb multi-word expression; andapplying the trained machine learning model to the distinguished sentence, the natural language preprocessing results for the distinguished sentence, and the constituent model results to obtain a prediction identifying one or more multi-word expressions occurring in the distinguished sentence, and, for each multi-word expression identified as occurring in the sentence, an indication of whether the multi-word expression is a noun multi-word expression or a verb multi-word expression.

6. The method of claim 5, further comprising: receiving an indication that a specified multi-word expression identified by the obtained prediction in the distinguished sentence is an invalid multi-word expression;in response to receiving the indication: constructing an exclusion training observation for the distinguished sentence that comprises: independent variable values comprising: the distinguished sentence;the natural language preprocessing results for the distinguished sentence; andthe constituent model result obtained for the distinguished sentence by each of the plurality of constituent models; anddependent variable values comprising: the training example's information identifying one or more multi-word phrases occurring in the sentence from which the specified multi-word expression is excluded; andthe training example's indications, for each multi-word phrase identified as occurring in the sentence, of whether the multi-word phrase is a noun multi-word expression or a verb multi-word expression; andusing the constructed training observation to train the machine learning model to predict dependent variable values based on independent variable values.

7. One or more instances of computer-readable media collectively having contents configured to cause a computing system to perform a method, none of the instances of computer-readable media constituting a transitory signal per se, the method comprising: receiving a sentence;for each of a plurality of constituent models, invoking the constituent model against the received sentence for the received sentence to obtain a constituent model result for the received sentence that identifies one or more portions of the received sentence each as a multi-word expression and specifies whether each multi-word expression is a noun multi-word expression or a verb multi-word expression; andapplying a trained machine learning model to the received sentence and the constituent model results to obtain a prediction identifying one or more multi-word expressions occurring in the received sentence, and, for each multi-word expression identified as occurring in the sentence, an indication of whether the multi-word expression is a noun multi-word expression or a verb multi-word expression.

8. The one or more instances of computer-readable media of claim 7, wherein the prediction obtained by the applying further identifies at least one portion of the received sentence identified by a constituent model as recommended for addition to a dictionary.

9. A method in a computing system, comprising: initializing a machine learning model;accessing a plurality of first natural language segments;for each of the plurality of accessed first natural language segments: accessing one or more indications, each indication identifying a contiguous sequence of words in the natural language segment;constructing a first model training observation in which the natural language segment is an independent variable, and the one or more indications are dependent variables; andusing the constructed first model training observations to train the initialized machine learning model to predict the set of multi-word expressions occurring in a subject natural language segment.

10. The method of claim 9, further comprising: accessing a subject natural language segment; andapplying the trained machine learning model to the accessed subject natural language segment to obtain a multi-word identification result for the subject natural language segment.

11. The method of claim 9 wherein at least one indication for at least one of the first plurality of natural language segments was available for access as a result of its automatic determination.

12. The method of claim 9 further comprising: receiving input specifying a contiguous sequence of words as not constituting a multi-word expression;selecting among the plurality of first natural language segments those natural language segments that include the specified contiguous sequence of words for each of the selected natural language segments: accessing one or more indications, each indication identifying a contiguous sequence of words in the natural language segment;removing from the accessed indications any indications identifying the specified contiguous sequence of words;constructing a second model training observation in which the natural language segment is an independent variable, and any indications among the one or more indications that are not removed are dependent variables; andusing the constructed first model training observations to retrain the trained machine learning model.

13. The method of claim 12, further comprising: accessing a subject natural language segment; andapplying the retrained machine learning model to the accessed subject natural language segment to obtain a multi-word identification result for the subject natural language segment.

14. One or more computer memories collectively having contents configured to cause a computing system to perform a method, the method comprising: receiving a natural language segment;comparing the natural language segment to a dictionary specifying multi-word expressions to identify one or more first contiguous sequences of words in the natural language segment that are each specified by the dictionary;for each of one or more of the first contiguous sequences of words, constructing one or more candidate sequences of words, each of the constructed candidate sequences of words comprising one of the first contiguous sequences of words plus one or more adjacent words in the natural language segment;evaluating each candidate sequence of words;based on evaluation, selecting at least one of the candidate sequences of words; andoutputting a set of multi-word expressions for the natural language segment that comprises the first contiguous sequences of words and the selected candidate sequences of words.

15. The one or more computer memories of claim 14, the method further comprising: causing the dictionary to be modified to specify each of the selected candidate sequences of words as a multi-word expression, to obtain a first modified dictionary.

16. The one or more computer memories of claim 15, the method further comprising: receiving a second natural language segment; andrepeating the comparing, constructing, evaluating, selecting, and outputting for the second natural language segment using the first modified dictionary.

17. The one or more computer memories of claim 16, the method further comprising: receiving input indicating that a distinguished contiguous sequence of words is not a multi-word expression; andin response to the receiving, causing the first modified dictionary to be modified to no longer specify the distinguished contiguous sequence of words as a multi-word expression, to obtain a second modified dictionary.

18. The one or more computer memories of claim 15, the method further comprising: receiving a third natural language segment; andrepeating the comparing, constructing, evaluating, selecting, and outputting for the third natural language segment using the second modified dictionary.