Patent Application
20190237061
Machine learning techniques allow computing devices to make decisions or perform operations based on models that can be trained from examples. “Supervised” machine learning relies on a body of training data that is often manually labeled. Based on the labeled training data, a mathematical model can be built that allows the computing device to perform operations on novel data not found in the body of training data.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
A method for generating training data for training a natural language processing system comprises loading, into a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events. As used here, a dialogue event includes a client utterance or a computerized assistant response. The method further comprises acquiring a computer-readable command parametrized by a seed semantic parameter and describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript. The method further comprises acquiring an alternative semantic parameter differing from the seed semantic parameter, and re-parametrizing the computer-readable command by replacing the seed semantic parameter with the alternative semantic parameter. The method further comprises generating an alternative ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter, the alternative ordered subsequence of one or more dialogue events differing from the exemplary ordered subsequence of one or more dialogue events. The method further comprises outputting, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the computerized assistant.
Computerized assistants may include logic that is configured to respond to natural language queries or commands. For instance, a computerized assistant may be configured to respond to a weather query by accessing a remote weather service and audibly disclosing information pertaining to the query using natural language. This is illustrated in
Because there are any number of ways that a human user may choose to phrase a particular command or query, computerized assistants are often trained using machine learning. For example, supervised machine learning may be used to build a body of training data including various client utterances (e.g., natural language queries or commands), as well as manually-provided annotations that define computer-readable interpretations of the client utterances. Client utterances include any combinations of human language words on which a computerized assistant may be trained, and may be human-generated (e.g., written or spoken) or computer-generated. Through annotation of such utterances, a model can be built that allows a computerized assistant to respond to novel utterances that were not included in the body of training data.
To achieve acceptable results using this approach, it is typically necessary to accumulate large numbers (e.g., thousands or millions) of annotated utterances to build the body of training data. As such, annotation of training inputs may be crowdsourced to large groups of human annotators. Furthermore, annotation of training data is a technical task that frequently requires annotator expertise, e.g., a background in computer programming to properly perform. This requirement limits the pool of individuals who can serve as annotators, which makes it more difficult to accumulate a suitable body of training data.
Accordingly, the disclosed training pipeline may be used to automatically generate training data. The training pipeline includes machines for acquiring training data in the form of exemplary dialogues and annotations for the dialogues, as well as machines for automatically generating variants of the dialogues to expand coverage of the training data. The training pipeline may include a computing device that presents an utterance annotation interface through which example client utterances may be annotated in an intuitive and user-friendly manner. Specifically, the utterance annotation interface may include a transcript portion in which an unannotated transcript of one or more dialogue events is displayed. The dialogue events may include, for example, example client utterances (e.g., natural language commands or queries), as well as computerized assistant responses to such utterances. The utterance annotation interface may include an annotation portion that in turn includes a hierarchical menu of candidate utterance annotations. By interacting with this hierarchical menu, a human annotator having relatively little experience in machine learning may select utterance annotations that, for a given example client utterance, define a computer-readable interpretation of the utterance. An annotated utterance may then be output to a data store and used for training a computerized assistant.
In this manner, the herein-described training pipeline and utterance annotation interface improves over conventional computerized approaches for annotating training data which, as discussed above, typically require human annotators to have extensive expertise and/or or programming experience. As a practical result, a body of training data for training a computerized assistant (or other application that relies on machine learning) may be more quickly and efficiently accumulated. This in turn improves the performance of the computerized assistant.
Client devices 102A and 102B each run a computerized assistant application 104A/104B. As used herein, a “computerized assistant application” includes any suitable combination of software, hardware, and/or firmware components usable to perform any or all of the functions of a computerized assistant, including detecting and responding to client utterances. For example, a computerized assistant application may take the form of a user-installable software application, a component of a device operating system, a web application accessed via an Internet browser, an Application Specific Integrated Circuit (ASIC), etc. Furthermore, a computerized assistant application may run entirely locally, and/or may communicate/cooperate with one or more remote devices or services and/or other aspects of the computerized assistant application.
This scenario is illustrated in
In any case, in the illustrated example, initial training of the computerized assistant at least partially occurs at server 206. As discussed above, annotated client utterances may be used to develop and refine a model via supervised machine learning that allows the computerized assistant to respond to novel client utterances. An utterance annotation computing device 202 is configured to present an utterance annotation interface 204 by which a human annotator may annotate client utterances. The annotated utterances may then be output to data store 208, at which point they may be used to train the computerized assistant.
In some examples, one or more of the devices shown in
Pipeline 300 is shown in more detail in
Pipeline 300 includes a deployed computerized assistant machine 305. Deployed computerized assistant machine 305 is configured to provide computerized assistant functionality, e.g., based on a machine learning model learned from training data. Computerized assistant machine 305 may include any suitable machine learning technologies and/or natural language processing technologies, as will be described further with regard to
Deployed computerized assistant machine 305 is trained via machine learning training techniques at training machine 304. Training machine 304 may use any suitable machine learning training techniques, as described further with regard to
Accordingly, pipeline 300 includes a seed dialogue acquisition machine 301 configured to acquire exemplary labelled data for training. Seed dialogue acquisition machine 301 may include a historical agent log store 301A, configured for accessing historical examples of interaction between a client and deployed computerized assistant machine 305. In addition to such historical data, seed dialogue acquisition machine 301 is configured to receive crowd sketch data at a crowd sketch interface 301B. Crowd sketch data includes interaction between the deployed computerized assistant machine 305 and crowd workers (e.g., exemplary client users, or annotator workers with experience annotating dialogue examples). For example, crowd sketches may include dialogues conducted between the computerized assistant machine 305 and a human annotator based on a validation model 306, so as to validate the quality and/or confidence of the validation model 306. Crowd sketch interface 301B may be configured for any suitable interaction and data-gathering based on such interaction, e.g., free-form conversations and gathering indicators of client satisfaction during the interaction for use as reinforcement signals in reinforcement learning. Seed dialogues may be represented in a computer-readable form including an ordered sequence of one or more utterances by the computerized assistant and/or client(s), and/or primitive actions by the computerized assistant (e.g., application-programming interface (API) calls to look up information, cause a real-world result, etc.).
Pipeline 300 further includes an annotation acquisition machine 302 configured to acquire annotations for the exemplary seed dialogues acquired at seed dialogue acquisition machine 301 (e.g., in the form of historical data at 301A and/or crowd sketches at 301B). Annotation acquisition machine 302 is configured to present one or more user interfaces for human annotators (e.g., clients and/or volunteer or professional workers with any suitable amount of experience/training) to provide annotations for labelling exemplary seed dialogues. Annotations for exemplary seed dialogues may take any suitable form. For example, as will be described with regard to
Computer-readable labels (e.g., computer-readable commands or any other suitable labels) may represent semantic contents of a dialogue in a form that can be used by a deployed computerized assistant machine 305 for training and/or future decision-making. In a direct authoring interface 302B, a worker is presented with a seed dialogue and asked to hand-write an annotation command describing the semantics of the seed dialogue, e.g., using a library of domain-specific combinator functions. In an interactive repair interface 302A, a worker is presented with a seed dialogue and a selection of candidate annotations for the seed dialogue and is tasked with selecting a best annotation or indicating that none of the annotations are appropriate, thereby associating the most appropriate annotation (if any) with the seed dialogue. If there is no appropriate annotation, interactive repair interface 302A may be configured to present a direct authoring interface 302B as a fallback. Exemplary user interfaces for performing interactive repair and/or direct authoring are described further with regards to
Seed dialogue acquisition machine 301 and annotation acquisition machine 302 may obtain a plurality of different annotated seed dialogues, suitable for training at training machine 304. However, even if numerous examples are collected (e.g., from numerous client interactions with the system, and numerous human annotators interacting via crowd sketches, interactive repair, and direct authoring), even more data may be desired for training (e.g., the collected examples may be insufficient to obtain a deployed computerized assistant machine 305 that is able to handle diverse interactions not seen in the training data). Accordingly, pipeline 300 further includes a synthetic data generation machine 303 configured to automatically expand the training data to cover more different semantic situations and/or utterance phrasings. Synthetic data generation machine 303 optionally includes a coverage expansion machine 303A configured to expand an annotation command to cover other semantic situations by re-parametrizing the annotation command, and to generate a corresponding alternative dialogue to be paired with the annotation command to make a new annotated dialogue for training. Synthetic data generation machine 303 optionally further includes a paraphrase expansion machine 303B configured to generate alternative paraphrased dialogues to produce annotated dialogues that cover other wordings/phrasings than seen in the original annotated data acquired by machines 301 and 302. In some examples, coverage expansion machine 303A and/or paraphrase expansion machine 303B are configured to acquire new parameters/paraphrases from human annotators, e.g., as described with regard to
Synthetic data generation machine 303 may output a large plurality of training data (e.g., a substantially larger number of annotated dialogues than were acquired at seed dialogue acquisition machine 301 and annotation acquisition machine 302). Accordingly, this training data may be used for training at training machine 304, thereby continually updating deployed computerized assistant machine 305 to handle new situations exemplified in the generated training data. In some examples, the synthetic data may be of unknown quality (e.g., paraphrases generated from a thesaurus, natural language model, and/or from human annotators may be of unknown accuracy relative to original phrases). Accordingly, by updating validation model 306 and operating deployed computerized assistant machine 305 according to the validation model 306, the computerized assistant may be used for live interaction in new situations covered by the generated training data. Pipeline 300 is configured to continually improve functionality of the computerized assistant based on the new situations in live interaction, by generating historical agent data and/or crowd sketches based on these new situations, and acquiring annotations for these new situations for further data generation and/or training. By expanding the model to handle a large number of previously-unseen situations from generated data, the computerized assistant may be useable for interaction in a wide variety of situations.
At 404, method 400 includes acquiring a computer-readable command parametrized by a seed semantic parameter and describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript. Acquiring a computer-readable command for the computer-readable transcript may be performed in any suitable manner, e.g., using direct authoring or interactive repair in an annotation acquisition machine 302 (described further below with regard to
At 406, method 400 includes acquiring an alternative semantic parameter differing from the seed semantic parameter, and re-parametrizing the computer-readable command by replacing the seed semantic parameter with the alternative semantic parameter. The alternative semantic parameter for replacing the seed semantic parameter may be acquired in any suitable fashion, for example, using a user interface for supplying alternative semantic parameters as described below with regard to
At 408, method 400 further includes generating an alternative ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter, the alternative ordered subsequence of one or more dialogue events differing from the exemplary ordered subsequence of one or more dialogue events. Generating the alternative ordered subsequence is based on executing semantic contents of the computer-readable command, e.g., sub-commands to detect user utterances, output computer assistant utterances, and/or perform primitive actions, as will be described further below with regard to
At 410, method 400 further includes outputting, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the computerized assistant. For example, the alternative computer-readable transcript may represent the ordered sequence of user utterances to be detected, and computer utterances and/or actions in response to the user utterances, in the same format as historical agent logs collected during system operation, which in turn may be useable for training.
At 502, method 500 includes loading, into a computer memory, a computer-readable seed command describing an ordered sequence of two or more top-level dialogue events. As indicated at 504, a dialogue event includes a client utterance or a computerized assistant response. As indicated at 506, the seed command includes one or more sub-commands. Each sub-command of a seed command may correspond to a portion of the ordered sequence of two or more top-level dialogue events. As indicated at 508, a focal sub-command of the one or more sub-commands is parametrized by a seed semantic parameter. For example,
Returning to
Returning to
At 514, method 500 further includes, for each of the plurality of different synthetic semantic parameters: saving a corresponding re-parametrized focal sub-command. For example, as shown in
In some examples, a focal sub-command is configured to generate up to one result dialogue event, and a second one of the sub-commands is configured to be conditionally executed, responsive to the focal sub-command returning a result dialogue event, the method further including re-parametrizing the second sub-command. For example, as shown in
Returning briefly to
In some examples, re-parametrizing the command includes operating a computer model linking a computer-readable description of the seed semantic parameter to one or more candidate alternative semantic parameters, in order to select a replacement parameter for the command (or for a focal sub-command, multiple sub-commands, etc.). For example, the computer model may include a natural language model, a neural network, a dictionary/thesaurus, and/or a type system configured to categorize semantic features in different categories (e.g., times, places) or based on relationships (e.g., synonyms, antonyms, etc.).
Alternately or additionally, different synthetic semantic parameters may be acquired via a user interface for acquiring synthetic semantic parameters from human annotators. The user interface is configured to graphically display a portion of the computer-readable transcript corresponding to the seed semantic parameter; and receive one or more computer inputs indicating the alternative semantic parameter. For example,
In some examples, as alternative semantic parameters are acquired for a given seed semantic parameter, they may be stored in a data store with an address defined by the seed semantic parameter. Accordingly, acquiring a semantic parameter may be based on looking up one or more candidate synthetic semantic parameters in the semantic parameter data store by locating an address defined by the seed semantic parameter, thereby avoiding replicated effort to acquire new alternative semantic parameters.
In some examples, generating the ordered sequence of one or more dialogue events includes: outputting an initial ordered sequence of one or more dialogue events based on the re-parametrized focal sub-command; and paraphrasing the initial ordered sequence to generate the ordered sequence. For example, as shown in
Alternately or additionally, in some examples, paraphrasing the initial ordered subsequence includes: graphically displaying a portion of the initial ordered subsequence; and receiving one or more computer inputs defining a candidate paraphrase of the portion. For example,
After acquiring a paraphrase for a transcript portion (e.g., from a human annotator or a computer model), the methods of the present disclosure may further include outputting, to a paraphrase data store, a computer-readable representation of the candidate paraphrase for a portion of the initial ordered subsequence, wherein the stored representation is addressable based on the portion of the initial ordered subsequence. Accordingly, paraphrasing the initial ordered subsequence may include, for a portion of the initial ordered subsequence, looking up one or more candidate paraphrases in the paraphrase data store by locating an address defined by the portion, to retrieve previously-acquired paraphrases (e.g., in lieu of gathering new data from human annotators).
As described above with regard to
As discussed above, use of an utterance annotation interface presented by an utterance annotation computing device may facilitate intuitive and user-friendly annotation of client utterances.
At 1002, method 1000 includes graphically displaying an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, with at least one of the dialogue events taking the form of an example client utterance. This is shown in
As shown, interface 1100 includes a transcript portion 1102 and an annotation portion 1104. The transcript portion includes an unannotated transcript 1106 including several dialogue events 1108A-C represented as text, one of which is an example client utterance 1110. Dialogue events include any or all natural language interactions between a client and a computerized assistant, such as client utterances and computerized assistant responses. Furthermore, as discussed above, a client utterance may include any combination of human language words based on which a computerized assistant may be trained, and such utterances may be specified by a human or computer-generated. As used herein, an “example client utterance” is any client utterance displayed in the transcript portion that is annotated or will be annotated by a human annotator.
The unannotated transcript, as well as the included dialogue events, may originate from any suitable source. In one example, one or more dialogue events (e.g., client utterances) in the unannotated transcript may be written by the human annotator. Additionally, or alternatively, all or part of the unannotated transcript may be retrieved from a database of unannotated example client utterances. Such a database may be stored locally, retrieved from a remote source such as server 206 or data store 208 of
Returning briefly to
As discussed above, in supervised machine learning, training data is often manually labeled by human annotators. This is often a specialized and technical task, requiring the human annotators to have advanced expertise and/or a background in computer programming. By contrast, the utterance annotation interface described herein simplifies the process of utterance annotation such that it can be performed by human annotators having relatively less experience, thus greatly broadening the pool of annotators that can be used to generate the training data. This is achieved by providing a hierarchical menu that includes a plurality of candidate utterance annotations. A “candidate utterance annotation” can be described as a human-readable label that, when selected, specifies a computer-readable label for an example client utterance. Annotating an example client utterance therefore becomes a matter of selecting, from a hierarchical menu, a set of human-readable utterance annotations, in much the same manner as filling out a computerized form. The set of selected utterance annotations define a computer-readable interpretation of the example client utterance, allowing for training of the computerized assistant.
An example workflow for using an utterance annotation interface is illustrated in
As shown, data in the hierarchical menu is arranged in a hierarchical tree structure 1113. The highest level of the hierarchical menu illustrated in
In some cases, the annotation utterance computing device may maintain relationships or links between various leaves or nodes in the hierarchical menu. For instance, while two utterance annotations may be represented as leaves on two different branches in the tree structure, the two utterance annotations may nonetheless be linked such that they may, in various instances, be graphically displayed with each other, recommended together, automatically applied together, etc. In one example scenario, after a human annotator selects a “current time” utterance annotation, a “current place” utterance annotation that is hierarchically nested under a different category-specific domain or subdomain may be automatically suggested based on a preexisting link between the “current time” and “current place” annotations.
With reference now to
Furthermore, in some examples, each of the category-specific domains and subdomains may themselves constitute candidate utterance annotations. In other words, each selection of a category-specific domain or subdomain may provide high-level information about the example client utterance that can be used by a computerized assistant to respond to similar utterances in the future. In the illustrated example, selection of the “weather” category-specific domain is recorded as an annotation and displayed in annotation portion 1104 in
Turning now to
Returning briefly to
Turning now to
It will be understood that the specific syntax and format of the computer-readable interpretation of the example client utterance will vary depending on the implementation. Furthermore, the specific example shown in
Although
After receiving the computer inputs specifying the utterance annotations, the utterance annotation computing device may in some cases display a computerized assistant response to the example client utterance based on the one or more utterance annotations. This is also shown in
In some situations, the utterance annotation computing device may receive computer inputs that specify unsuitable utterance annotations for a particular client utterance. For example, a novice or distracted human annotator may inadvertently select inappropriate utterance annotations that collectively define an inaccurate computer-readable interpretation of the example client utterance. Accordingly, displaying a computerized assistant response to the example client utterance that is generated based on the selected utterance annotations, as is shown in
To mitigate the risk of a human annotator providing incorrect utterance annotations, the utterance annotation computing device may in some cases be configured to provide a modified, training-mode version of the utterance annotation interface. Such an interface may operate in a substantially similar manner as described above with respect to
Returning again to
Turning now to
For example,
As indicated above, the unannotated transcript displayed in the transcript portion of the utterance annotation interface may include any number of dialogue events in an ordered sequence and may include any number of example client utterances for annotation. For example, in addition to a first example client utterance as described above, a dialogue event in the unannotated transcript may take the form of a second example client utterance that occurs later in the ordered sequence. In some scenarios, the second example client utterance may refer back to (e.g., modify) the first example client utterance. This is illustrated in
Accordingly, the utterance annotation computing device may receive, for one or more response parameters in the second example client utterance (e.g., “the next day”), one or more computer inputs selecting one or more utterance annotations from the hierarchical menu that collectively define a computer-readable interpretation of the second example client utterance. The utterance annotations selected for the second example client utterance may in some cases modify the one or more utterance annotations selected for the first example client utterance, such as when the second example client utterance refers back to or modifies the first example client utterance. This is also shown in
Thus far, the computerized assistant responses to the example client utterances discussed herein have been computer-generated, either by the utterance annotation computing device or by other devices as discussed above. However, in some implementations, all or part of a computerized assistant response to a particular client utterance may be manually specified via computer inputs to the utterance annotation computing device—for instance, by a human annotator who selected utterance annotations for the client utterance.
This is illustrated in
Furthermore, in some cases, the computing device may automatically generate multiple candidate portions of the computerized response, and the computing device may receive one or more computer inputs that select from among the multiple candidate portions. This is illustrated in
The methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as an executable computer-application program, a network-accessible computing service, an application-programming interface (API), a library, or a combination of the above and/or other compute resources. A variety of different computer languages, compilers, data structures, etc. may be used. For example, computer-readable commands (e.g., utterance annotations) may be stored and/or accessed in the form of any suitable computer data structure for representing programming language syntax, control and/or data flow, and/or hierarchical structure (e.g., graph data structures, tree data structures, etc.). In some examples, the computer-readable command may be a plain-text command with text content adhering to a pre-defined syntax. In some examples, the computer-readable command may be stored as a JavaScript Object Notation (JSON) file. In some examples, the computer-readable command may be defined in a syntax supporting variable parameter names. In some examples, the computer-readable command may be defined to include one or more sub-commands.
Semantic parameters may be text or any other suitable structured data (e.g., binary data) for representing semantic values that may be relevant to the semantics of a dialogue described by a command. Semantic parameters may be defined using primitive data types (e.g., text, floating point number, integer), enumerative data types such as pre-defined lists of color names or airport names, and/or any other structured data that may be represented in the command language syntax (e.g., in JSON). Computer-readable commands may have input/output types determined by types of semantic parameters and/or sub-commands, e.g., a computer-readable command may have an input/output type indicating that it requires a parameter indicating a type of weather, and that it outputs a Boolean value.
In some examples, computer-readable commands, sub-commands, and/or semantic parameters may be given names, so as to be stored/accessed in an indexed fashion. For example, sub-commands, and/or semantic parameters may be included in multiple different commands. Commonly-included sub-commands and/or semantic parameters may be stored only once and referred to by name within commands that include them. Accordingly, storing commands, sub-commands and/or semantic parameters may improve efficiency of storage and retrieval for computer-readable commands. In some examples, commands and/or parameters may be additionally or alternatively indexed based on input/output types and parameter types. Accordingly, commands and/or parameters may be retrieved based on matching the input/output types and parameter types. Indexing commands and/or parameters based on input/output types may facilitate efficient retrieval of alternative commands that may pertain to a dialogue (e.g., by retrieving a plurality of commands that have a same or similar input/output type), efficient retrieval of alternative semantic parameters having a common type (e.g., for retrieving alternative semantic parameters that could be used to replace a semantic parameter). In some examples, commands may be indexed based on domain metadata indicating a domain of usage (e.g., “weather,” “restaurants,” and the like). Name, type, and/or domain metadata may facilitate efficient retrieval of relevant computer-readable commands, sub-commands, and/or semantic parameters, for example: for presenting commands to a human annotator during interactive repair; for finding relevant commands and/or sub-commands to present to a human annotator for constructing an annotation for an utterance; for generating alternative semantic parameters for replacing a seed semantic parameter to generate annotated utterances for use as training data; etc.
Dialogue transcripts, transcript portions, and individual utterances may be stored/accessed in the form of any suitable computer data structure, e.g., plain-text or JSON. For example, individual utterances may be stored as plain-text representing natural language content. Transcript portions and/or dialogue transcripts include one or more individual utterances in an ordered sequence. In general, transcript portions may be represented using the same data structures and the following description regarding dialogue transcripts also applies to transcript portions. Dialogue transcripts may include multiple individual utterances in the form of a collection data structure, concatenated sequence of utterances separated by delimiters, etc. Utterances may be marked as arising from a computerized assistant or client. In some examples, a transcript may include interaction by more than two entities (e.g., a computerized assistant and two different clients), and accordingly the transcript may be marked to indicate speaker identity for each utterance. In some examples, dialogue transcripts may include time-stamps or other time information indicating a temporality of the individual utterances. Optionally, in some examples, in addition to utterances, a transcript may further indicate actions performed by a computerized assistant, other inputs to the computerized assistant (e.g., camera input, keyboard input, etc.), or any other suitable information relating to the dialogue. Whether stored as plain-text, JSON, or another data structure, dialogue transcripts and/or transcript portions may be displayed as an organized/ordered sequence of utterances. The displayed sequence of utterances may be laid-out and/or annotated (e.g., color coded) in any suitable fashion to indicate speaker identity for each utterance. Even when a dialogue transcript includes information other than utterances (e.g., computer-readable commands, time-stamps, etc.), optionally when displaying the utterance, such other information may be omitted so that only the utterances are displayed.
Although the present disclosure primarily describes annotation with regard to single utterances, the techniques disclosed herein may be similarly applied to acquire annotations for transcript portions or whole dialogue transcripts. Accordingly, as used herein, “annotated utterance” may refer to a single annotated utterance, as well as to an annotated transcript of one or more utterances. A computer-readable label (e.g., one or more utterance annotations) specified by a human annotator may be associated with any suitable portion of a dialogue transcript and/or with any single utterance within a dialogue transcript, to indicate how the label pertains to that specific portion/utterance. Accordingly, acquiring a computer-readable label from a human annotator may include storing a new annotated utterance linking the utterance annotation to a particular transcript portion/utterance that the human annotator was shown during annotation. Accordingly, each computer-readable label may be stored/accessed/retrieved in association with the particular utterances.
An annotated utterance includes one or more utterances, along with an annotation in the form of a computer-readable label useable for machine learning training. In some examples, the computer-readable label is a computer-readable command as described above. Annotated utterances and/or annotated dialogue transcripts may be stored in any suitable format. In some examples, annotated utterances may be represented by storing one or more computer-readable labels in an indexed fashion with an index based on one or more utterances (e.g., a hash value of the one or more utterances). Accordingly, the associated dialogue transcript may be used as an index to look up the computer-readable labels for the one or more utterances, thereby facilitating efficient retrieval of labels. Alternately or additionally, annotated utterances may be stored as a collection of annotation-utterance pairs, wherein each pair includes a dialogue transcript component and a label component, and wherein the pairs are arranged in any suitable collection data structure. For example, when the computer-readable label is a computer-readable command, an annotated utterance may be represented as a JSON data structure including two JSON data structure components: 1) a dialogue transcript JSON data structure; and 2) a computer-readable command JSON data structure; similarly, a plurality of annotated utterances may be represented as a JSON data structure including an array containing a plurality of annotated utterance elements.
Computing system 1500 includes a logic subsystem 1502 and a storage subsystem 1504. Computing system 1500 may optionally include a display subsystem 1506, input subsystem 1508, communication subsystem 1510, and/or other subsystems not shown in
Logic subsystem 1502 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions that are part of one or more applications, services, or other logical constructs. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic subsystem optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by remotely-accessible, networked computing devices configured in a cloud-computing configuration.
Storage subsystem 1504 includes one or more physical devices configured to temporarily and/or permanently hold computer information such as data and instructions executable by the logic subsystem. When the storage subsystem includes two or more devices, the devices may be collocated and/or remotely located. Storage subsystem 1504 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices. Storage subsystem 1504 may include removable and/or built-in devices. When the logic subsystem executes instructions, the state of storage subsystem 1504 may be transformed—e.g., to hold different data.
Aspects of logic subsystem 1502 and storage subsystem 1504 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
The logic subsystem and the storage subsystem may cooperate to instantiate one or more logic machines. As used herein, the term “machine” is used to collectively refer to hardware and any software, instructions, and/or other components cooperating with such hardware to provide computer functionality. In other words, “machines” are never abstract ideas and always have a tangible form. A machine may be instantiated by a single computing device, or a machine may include two or more sub-components instantiated by two or more different computing devices. In some implementations a machine includes a local component (e.g., software application) cooperating with a remote component (e.g., cloud computing service). The software and/or other instructions that give a particular machine its functionality may optionally be saved as an unexecuted module on a suitable storage device. For example, machines instantiated by the logic subsystem and storage subsystem include the machines shown in pipeline 300, e.g., seed dialogue acquisition machine 301, annotation acquisition machine 302, synthetic data generation machine 303, training machine 304, and deployed computerized assistant machine 305.
Machines may be implemented using any suitable combination of state-of-the-art and/or future machine learning (ML), artificial intelligence (AI), and/or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated in an implementation of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and/or videos, temporal convolutional neural networks for processing audio signals and/or natural language sentences, and/or any other suitable convolutional neural networks configured to convolve and pool features across one or more temporal and/or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom Filters, Neural Turing Machine and/or Neural Random Access Memory), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and/or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and/or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and/or AI knowledge bases), and/or natural language processing techniques (e.g., tokenization, stemming, constituency and/or dependency parsing, and/or intent recognition, segmental models, and/or super-segmental models (e.g., hidden dynamic models)).
In some examples, the methods and processes described herein may be implemented using one or more differentiable functions, wherein a gradient of the differentiable functions may be calculated and/or estimated with regard to inputs and/or outputs of the differentiable functions (e.g., with regard to training data, and/or with regard to an objective function). Such methods and processes may be at least partially determined by a set of trainable parameters. Accordingly, the trainable parameters for a particular method or process may be adjusted through any suitable training procedure, in order to continually improve functioning of the method or process.
Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., deep Q learning based on feedback) and/or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood methods, and/or expectation maximization. In some examples, a plurality of methods, processes, and/or components of systems described herein may be trained simultaneously with regard to an objective function measuring performance of collective functioning of the plurality of components (e.g., with regard to reinforcement feedback and/or with regard to labelled training data). Simultaneously training the plurality of methods, processes, and/or components may improve such collective functioning. In some examples, one or more methods, processes, and/or components may be trained independently of other components (e.g., offline training on historical data).
The methods and processes disclosed herein may be implemented using one or more language models. Language models may utilize vocabulary features to guide sampling/searching for words for recognition of speech. For example, a language model may be at least partially defined by a statistical distribution of words or other vocabulary features. For example, a language model may be defined by a statistical distribution of n-grams, defining transition probabilities between candidate words according to vocabulary statistics. The language model may be further based on any other appropriate statistical features, and/or results of processing the statistical features with one or more machine learning and/or statistical algorithms (e.g., confidence values resulting from such processing). In some examples, a statistical model may constrain what words may be recognized for an audio signal, e.g., based on an assumption that words in the audio signal come from a particular vocabulary.
Alternately or additionally, the language model may be based on one or more neural networks previously trained to represent audio inputs and words in a shared latent space, e.g., a vector space learned by one or more audio and/or word models (e.g., wav2letter and/or word2vec). Accordingly, finding a candidate word may include searching the shared latent space based on a vector encoded by the audio model for an audio input, in order to find a candidate word vector for decoding with the word model. The shared latent space may be utilized to assess, for one or more candidate words, a confidence that the candidate word is featured in the speech audio.
The language model may be used in conjunction with an acoustical model configured to assess, for a candidate word and an audio signal, a confidence that the candidate word is included in speech audio in the audio signal based on acoustical features of the word (e.g., mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, the language model may incorporate the acoustical model (e.g., assessment and/or training of the language model may be based on the acoustical model). The acoustical model defines a mapping between acoustic signals and basic sound units such as phonemes, e.g., based on labelled speech audio. The acoustical model may be based on any suitable combination of state-of-the-art or future machine learning (ML) and/or artificial intelligence (AT) models, for example: deep neural networks (e.g., long short-term memory, temporal convolutional neural network, restricted Boltzmann machine, deep belief network), hidden Markov models (HMM), conditional random fields (CRF) and/or Markov random fields, Gaussian mixture models, and/or other graphical models (e.g., deep Bayesian network). Audio signals to be processed with the acoustic model may be pre-processed in any suitable manner, e.g., encoding at any suitable sampling rate, Fourier transform, band-pass filters, etc. The acoustical model may be trained to recognize the mapping between acoustic signals and sound units based on training with labelled audio data. For example, the acoustical model may be trained based on labelled audio data comprising speech audio and corrected text, in order to learn the mapping between the speech audio signals and sound units denoted by the corrected text. Accordingly, the acoustical model may be continually improved to improve its utility for correctly recognizing speech audio.
In some examples, in addition to statistical models, neural networks, and/or acoustical models, the language model may incorporate any suitable graphical model, e.g., a hidden Markov model (HMM) or a conditional random field (CRF). The graphical model may utilize statistical features (e.g., transition probabilities) and/or confidence values to determine a probability of recognizing a word, given the speech audio and/or other words recognized so far. Accordingly, the graphical model may utilize the statistical features, previously trained machine learning models, and/or acoustical models to define transition probabilities between states represented in the graphical model.
The methods and processes disclosed herein may be configured to give users and/or any other humans control over any private and/or potentially sensitive data. Whenever data is stored, accessed, and/or processed, the data may be handled in accordance with privacy and/or security standards. When user data is collected, users or other stakeholders may designate how the data is to be used and/or stored. Whenever user data is collected for any purpose, the user owning the data should be notified, and the user data should only be collected when the user provides affirmative consent. If data is to be collected, it can and should be collected with the utmost respect for user privacy. If the data is to be released for access by anyone other than the user or used for any decision-making process, the user's consent may be collected before using and/or releasing the data. Users may opt-in and/or opt-out of data collection at any time. After data has been collected, users may issue a command to delete the data, and/or restrict access to the data. All potentially sensitive data optionally may be encrypted and/or, when feasible anonymized, to further protect user privacy. Users may designate portions of data, metadata, or statistics/results of processing data for release to other parties, e.g., for further processing. Data that is private and/or confidential may be kept completely private, e.g., only decrypted temporarily for processing, or only decrypted for processing on a user device and otherwise stored in encrypted form. Users may hold and control encryption keys for the encrypted data. Alternately or additionally, users may designate a trusted third party to hold and control encryption keys for the encrypted data, e.g., so as to provide access to the data to the user according to a suitable authentication protocol.
When the methods and processes described herein incorporate ML and/or AI components, the ML and/or AI components may make decisions based at least partially on training of the components with regard to training data. Accordingly, the ML and/or AI components can and should be trained on diverse, representative datasets that include sufficient relevant data for diverse users and/or populations of users. In particular, training data sets should be inclusive with regard to different human individuals and groups, so that as ML and/or AI components are trained, their performance is improved with regard to the user experience of the users and/or populations of users.
For example, a dialogue system according to the present disclosure may be trained to interact with different populations of users, using language models that are trained to work well for those populations based on language, dialect, accent, and/or any other features of speaking style of the population.
ML and/or AI components may additionally be trained to make decisions so as to minimize potential bias towards human individuals and/or groups. For example, when AI systems are used to assess any qualitative and/or quantitative information about human individuals or groups, they may be trained so as to be invariant to differences between the individuals or groups that are not intended to be measured by the qualitative and/or quantitative assessment, e.g., so that any decisions are not influenced in an unintended fashion by differences among individuals and groups.
ML and/or AI components can and should be designed to provide context as to how they operate as much as is possible, so that implementers of ML and/or AI systems can be accountable for decisions/assessments made by the systems. For example, ML and/or AI systems should have replicable behavior, e.g., when they make pseudo-random decisions, random seeds should be used and recorded to enable replicating the decisions later. As another example, data used for training and/or testing ML and/or AI systems should be curated and maintained to facilitate future investigation of the behavior of the ML and/or AI systems with regard to the data. Furthermore, ML and/or AI systems can and should be continually monitored to identify potential bias, errors, and/or unintended outcomes.
When included, display subsystem 1506 may be used to present a visual representation of data held by storage subsystem 1504. This visual representation may take the form of a graphical user interface (GUI). Display subsystem 1506 may include one or more display devices utilizing virtually any type of technology. In some implementations, display subsystem may include one or more virtual-, augmented-, or mixed reality displays.
When included, input subsystem 1508 may comprise or interface with one or more input devices. An input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition.
When included, communication subsystem 1510 may be configured to communicatively couple computing system 1500 with one or more other computing devices. Communication subsystem 1510 may include wired and/or wireless communication devices compatible with one or more different communication protocols. The communication subsystem may be configured for communication via personal-, local- and/or wide-area networks.
In an example, a method for generating training data for training a natural language processing system comprises: loading, into a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events, wherein a dialogue event includes a client utterance or a computerized assistant response; acquiring a computer-readable command parametrized by a seed semantic parameter and describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript; acquiring an alternative semantic parameter differing from the seed semantic parameter, and re-parametrizing the computer-readable command by replacing the seed semantic parameter with the alternative semantic parameter; generating an alternative ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter, the alternative ordered subsequence of one or more dialogue events differing from the exemplary ordered subsequence of one or more dialogue events; and outputting, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the natural language processing system. In this or any other example, the computer-readable command includes one or more sub-commands, each sub-command corresponding to a portion of the ordered sequence of one or more dialogue events. In this or any other example, re-parametrizing the computer-readable command further includes, for a focal sub-command of the one or more sub-commands, replacing a semantic parameter associated with the focal sub-command with a different, synthetic semantic parameter. In this or any other example, acquiring the computer-readable command includes: graphically displaying the computer-readable transcript; and receiving one or more computer inputs selecting the computer-readable command. In this or any other example, the method further comprises graphically displaying a hierarchical menu including a plurality of candidate commands, wherein the one or more computer inputs selecting the computer-readable command indicate one of the candidate commands. In this or any other example, acquiring the alternative semantic parameter includes: graphically displaying a portion of the computer-readable transcript corresponding to the seed semantic parameter; and receiving one or more computer inputs indicating the alternative semantic parameter. In this or any other example, acquiring the alternative semantic parameter includes operating a computer model linking a computer-readable description of the seed semantic parameter to one or more candidate alternative semantic parameters. In this or any other example, the method further comprises outputting, to a semantic parameter data store, a computer-readable representation of the alternative semantic parameter, addressable based on the seed semantic parameter. In this or any other example, acquiring the alternative semantic parameter includes looking up one or more candidate alternative semantic parameters in the semantic parameter data store by locating an address defined by the seed semantic parameter. In this or any other example, generating the alternative ordered subsequence of one or more dialogue events includes: outputting an initial ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter; and paraphrasing the initial ordered subsequence to generate the alternative ordered subsequence. In this or any other example, paraphrasing the initial ordered subsequence includes, for a portion of the initial ordered subsequence, operating a computer model to select a candidate paraphrase of the portion. In this or any other example, the computer model includes a natural language model configured to recognize a semantic relationship between a phrase and a candidate paraphrase for the phrase. In this or any other example, paraphrasing the initial ordered subsequence includes: graphically displaying a portion of the initial ordered subsequence; and receiving one or more computer inputs defining a candidate paraphrase of the portion. In this or any other example, the method further comprises outputting, to a paraphrase data store, a computer-readable representation of a candidate paraphrase for a portion of the initial ordered subsequence, addressable based on the portion of the initial ordered subsequence. In this or any other example, paraphrasing the initial ordered subsequence includes, for a portion of the initial ordered subsequence, looking up one or more candidate paraphrases in the paraphrase data store by locating an address defined by the portion.
In an example, a pipeline for generating training data for training a natural language processing system comprises: a seed dialogue acquisition machine configured to load, from a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events, wherein a dialogue event includes a client utterance or a computerized assistant response; an annotation acquisition machine configured to acquire a computer-readable command parametrized by a seed semantic parameter and describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript; a synthetic data generation machine configured to: generate an alternative semantic parameter differing from the seed semantic parameter, and to re-parametrize the computer-readable command by replacing the seed semantic parameter with the alternative semantic parameter; generate an alternative ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter, the alternative ordered subsequence of one or more dialogue events differing from the exemplary ordered subsequence of one or more dialogue events; and output, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the natural language processing system. In this or any other example, the pipeline further comprises a deployed computerized assistant machine configured to interact with a client and to store, into a historical agent log store of the computer memory, a computer-readable transcript describing the interaction. In this or any other example, the pipeline further comprises a training machine configured to train the deployed computerized assistant machine based at least on the alternative computer-readable transcript including the alternative ordered sequence of one or more dialogue events. In this or any other example, outputting the alternative ordered subsequence in the synthetic data generation machine includes: outputting an initial ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter; and paraphrasing the initial ordered subsequence to generate the alternative ordered subsequence.
In an example, a computer system comprises: a logic device; and a storage device configured to hold instructions executable by the logic device to: load, into a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events, wherein a dialogue event includes a client utterance or a computerized assistant response; acquire a computer-readable command parametrized by a seed semantic parameter and describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript; acquire an alternative semantic parameter differing from the seed semantic parameter, and re-parametrize the computer-readable command by replacing the seed semantic parameter with the alternative semantic parameter; generate an alternative ordered subsequence of one or more dialogue events based on the computer-readable command and the alternative semantic parameter, the alternative ordered subsequence of one or more dialogue events differing from the exemplary ordered subsequence of one or more dialogue events; and output, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the natural language processing system.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
This application claims priority to U.S. Provisional Patent Application Ser. No. 62/624,697, filed Jan. 31, 2018, the entirety of which is hereby incorporated herein by reference for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62624697 | Jan 2018 | US |
