Patent Grant
11562146
There is an ever-growing need in the art for improved natural language generation (NLG) technology that harnesses computers to process data sets and automatically generate narrative stories about those data sets. NLG is a subfield of artificial intelligence (AI) concerned with technology that produces language as output on the basis of some input information or structure, in the cases of most interest here, where that input constitutes data about some situation to be analyzed and expressed in natural language. Many NLG systems are known in the art that use template approaches to translate data into text. However, such conventional designs typically suffer from a variety of shortcomings such as constraints on how many data-driven ideas can be communicated per sentence, constraints on variability in word choice, and limited capabilities of analyzing data sets to determine the content that should be presented to a reader.
As technical solutions to these technical problems in the NLG arts, the inventors note that the assignee of the subject patent application has previously developed and commercialized pioneering technology that robustly generates narrative stories from data, of which a commercial embodiment is the QUILL™ narrative generation platform from Narrative Science Inc. of Chicago, Ill. Aspects of this technology are described in the following patents and patent applications: U.S. Pat. Nos. 8,374,848, 8,355,903, 8,630,844, 8,688,434, 8,775,161, 8,843,363, 8,886,520, 8,892,417, 9,208,147, 9,251,134, 9,396,168, 9,576,009, 9,697,197, 9, 697,492, 9,720,890, and 9,977,773, and U.S. patent application Ser. No. 14/211,444 (entitled “Method and System for Configuring Automatic Generation of Narratives from Data”, filed Mar. 14, 2014), Ser. No. 15/253,385 (entitled “Applied Artificial Intelligence Technology for Using Narrative Analytics to Automatically Generate Narratives from Visualization Data”, filed Aug. 31, 2016), Ser. No. 15/666,151 (entitled “Applied Artificial Intelligence Technology for Interactively Using Narrative Analytics to Focus and Control Visualizations of Data”, filed Aug. 1, 2017), Ser. No. 15/666,168 (entitled “Applied Artificial Intelligence Technology for Evaluating Drivers of Data Presented in Visualizations”, filed Aug. 1, 2017), and Ser. No. 15/666,192 (entitled “Applied Artificial Intelligence Technology for Selective Control over Narrative Generation from Visualizations of Data”, filed Aug. 1, 2017); the entire disclosures of each of which are incorporated herein by reference.
The inventors have further extended on this pioneering work with improvements in AI technology as described herein.
For example, the inventors disclose how AI technology can be used in combination with composable communication goal statements and an ontology to facilitate a user's ability to quickly structure story outlines in a manner usable by a narrative generation system without any need to directly author computer code.
Moreover, the inventors also disclose that the ontology used by the narrative generation system can be built concurrently with the user composing communication goal statements. Further still, expressions can be attached to objects within the ontology for use by the narrative generation process when expressing concepts from the ontology as text in a narrative story. As such, the ontology becomes a re-usable and shareable knowledge-base for a domain that can be used to generate a wide array of stories in the domain by a wide array of users/authors.
The inventors further disclose techniques for editing narrative stories whereby a user's editing of text in the narrative story that has been automatically generated can in turn automatically result in modifications to the ontology and/or a story outline from which the narrative story was generated. Through this feature, the ontology and/or story outline is able to learn from the user's edits and the user is alleviated from the burden of making further corresponding edits of the ontology and/or story outline.
The inventors further disclose how the narrative analytics that are linked to communication goal statements can employ a conditional outcome framework that allows the content and structure of resulting narratives to intelligently adapt as a function of the nature of the data under consideration.
Further still, the inventors also disclose how “analyze” communication goals can be supported by the system, including various examples of communication goal statements that drive the generation of narratives that express various ideas that are deemed relevant to a given analysis communication goal.
Through these and other features, example embodiments of the invention provide significant technical advances in the NLG arts by harnessing AI computing to improve how narrative stories are generated from data sets while alleviating users from a need to directly code and re-code the narrative generation system, thereby opening up use of the AI-based narrative generation system to a much wider base of users (e.g., including users who do not have specialized programming knowledge).
The example embodiments described herein further extend and innovate on the pioneering work described in the above-referenced and incorporated patent application serial numbers U.S. Pat. Nos. 9,576,009, 9,697,197, 9,697,492, 9,720,890, and 9,977,773, where explicit representations of communication goals are used by AI technology to improve how NLG technology generates narratives from data. With example embodiments described herein, AI technology is able to process a communication goal statement in relation to a data set in order to automatically generate narrative text about that data set such that the narrative text satisfies a communication goal corresponding to the communication goal statement. Furthermore, innovative techniques are disclosed that allow users to compose such communication goal statements in a manner where the composed communication goal statements exhibit a structure that promotes re-usability and robust story generation.
At step 102, a processor maps data within the data set to the parameters of the communication goal statement. The processor can also perform this step in response to user input as discussed below with respect to example embodiments.
At step 104, a processor performs NLG on the parameterized communication goal statement and the mapped data. The end result of step 104 is the generation of narrative text based on the data set, where the content and structure of the narrative text satisfies a communication goal corresponding to the parameterized communication goal statement.
While
It should be understood that steps 102 and 104, as well as steps 112 and 114, need not be performed in lockstep order with each other where step 102 (or 112) maps all of the data before the system progresses to step 104 (or step 114). These steps can be performed in a more iterative manner if desired, where a portion of the data is mapped at step 102 (or step 112), followed by execution of step 104 (or step 114) on that mapped data, whereupon the system returns to step 102/112 to map more data for subsequent execution of step 104/114, and so on.
Furthermore, it should be understood that a system that executes the process flows of
The previous example shows how an outline can be built by adding sections and parameterizing goals completely from scratch. The user is generally not expected to start from scratch, however. A narrative generation system instance will generally include a library of prebuilt components that users can utilize to more easily and quickly build out their outline. The narrative generation system's library provides access to previously parameterized and composed goals, subsections, sections, and even fully defined outlines. These re-usable components come fully parameterized, but can be updated or adjusted for the specific project. These changes are initially isolated from the shared library of components.
Components from the system's shared library can be used in two ways. First, a new project can be created from an entire project blueprint providing all aspects of a project already defined. This includes sample data, data views, the ontology, outline, sections, parameterized goals, and data mappings. Second, a user can pull in predefined components from the system's library ad hoc while building a new project. For example, when adding a section to an outline, the user can either start from scratch with an empty section or use a predefined section that includes a set of fully parameterized goals.
The system's library of components can be expanded by users of the platform through a mechanism that enables users to share components they have built. Once a component (outline, ontology, section, etc.) is shared, other users can then use them from the system's library in their own projects.
Composable Communication Goal Statements:
As shown by
As shown by
As shown by
Another example of a base communication goal statement is “Callout the Entity” 408 as shown by
It should be understood that the base communication goal statements shown by
The system can store data representative of a set of available base communication goal statements in a memory for use as a library. A user can then select from among this set of base communication goal statements in any of a number of ways. For example, the set of available base communication goal statements can be presented as a menu (e.g., a drop down menu) from which the user makes a selection. As another example, a user can be permitted to enter text in a text entry box. Software can detect the words being entered by the user and attempt to match those words with one of the base communication goal statements as would be done with auto-suggestion text editing programs. Thus, as a user begins typing the character string “Compa . . . ”, the software can match this text entry with the base communication goal statement of “Compare the Value to the Other Value” and select this base communication goal statement at step 300.
Returning to
Step 302 allows a user to use the existing ontology to support parameterization of a base communication goal statement. For example, if the ontology 320 includes an entity type of “Salesperson” that has an attribute of “Sales”, a user who is parameterizing base communication goal statement 402 can cause the processor to access the existing ontology 320 at step 304 to select “Sales of the Salesperson” from the ontology 320 at step 306 to thereby specify the parameter to be used in place of parameter placeholder 412 and thereby create a communication goal statement of “Present the Sales of the Salesperson”.
Also, if the existing ontology 320 does not include the parameters desired by a user, step 306 can operate by a user providing user input that defines the parameter(s) to be used for parameterizing the communication goal statement. In this situation, the processor in turn builds/updates the ontology 320 to add the parameter(s) provided by the user. For example, if the ontology 320 did not already include “Sales” as an attribute of the entity type “Salesperson”, steps 306-308 can operate to add a Sales attribute to the Salesperson entity type, thereby adapting the ontology 320 at the same time that the user is composing the communication goal statement. This is a powerful innovation in the art that provides significant improvement with respect to how artificial intelligence can learn and adapt to the knowledge base desired by the user for use by the narrative generation system.
At step 310, the processor checks whether the communication goal statement has been completed. If so, the process flow ends, and the user has composed a complete communication goal statement. However, if other parameters still need to be specified, the process flow can return to step 302. For example, to compose a communication goal statement from the base communication goal statement 406 of “Compare the Value to the Other Value”, two passes through steps 302-308 may be needed for the user to specify the parameters for use as the Value and the Other Value.
Also shown by
Also shown by
As with the base communication goal statements, it should be understood that a practitioner may choose to employ more, fewer, or different parameterized communication goal statements in a narrative generation system. For example, a parameterized Review communication goal statement could be “Review the weekly cash balance of the company over the year”, and a parameterized Explain communication goal statement could be “Explain the profit of the store in the month”.
Ontology Data Structure:
Attributes 330 can be represented by their own data structures within the ontology and can take the form of a direct attribute 330a and a computed value attribute 330b. A direct attribute 330a is an attribute of an entity type that can be found directly within a data set (e.g., for a data set that comprises a table of salespeople within a company where the salespeople are identified in rows and where the columns comprise data values for information such as the sales and sales territory for each salesperson, the attribute “sales” would be a direct attribute of the salesperson entity type because sales data values can be found directly within the data set). A computed value attribute 330b is an attribute of an entity type that is derived in some fashion from the data set. Continuing with the example above, a direct attribute for the salesperson entity type might be a percentage of the company's overall sales that were made by the salesperson. This information is not directly present in the data set but instead is computed from data within the data set (e.g., by summing the sales for all salespeople in the table and computing the percentage of the overall sales made by an individual salesperson).
Both the direct attributes 330a and computed value attributes 330b can be associated with metadata such as a type 340 (e.g., currency, date, decimal, integer, percentage, string, etc.), and a name 342. However, computed value attributes 330b can also include metadata that specifies how the computed value attribute is computed (a computation specification 348). For example, if a computed value attribute 330b is an average value, the computation specification 348 can be a specification of the formula and parameters needed to compute this average value.
Each entity type 322 may also comprise one or more characterizations 332. For example, a characterization 332 of a “salesperson” might be a characterization of how well the salesperson has performed in terms of sales (e.g., a good performer, an average performer, a poor performer). Characterizations can be represented by their own data structures 332 within the ontology. A characterization 332 can include metadata such as a name 360 (e.g., sales performance). Also, each characterization 332 can include a specification of the qualifications 364 corresponding to the characterization. These qualifications 364 can specify one or more of the following: (1) one or more attributes 330 by which the characterization will be determined, (2) one or more operators 366 by which the characterization will be determined, and (3) one or more value(s) 368 by which the characterization will be determined. For example, a “good performer” characterization for a salesperson can be associated with a qualification that requires the sales for the salesperson to exceed a defined threshold. With such an example, the qualifications 364 can take the form of a specified attribute 330 of “sales”, an operator 366 of “greater than”, and a value 368 that equals the defined threshold (e.g., $100,000).
Each entity type 322 may also comprise one or more relationships 334. Relationships 334 are a way of identifying that a relationship exists between different entity types and defining how those different entity types relate to each other. Relationships can be represented by their own data structures 334 within the ontology. A relationship 334 can include metadata such as the related entity type 350 with respect to the subject entity type 322. For example, a “salesperson” entity type can have a relationship with a “company” entity type to reflect that the salesperson entity type belongs to a company entity type. The ontological objects (e.g., entity types 322, direct attributes 330a, computed value attributes 330b, characterizations 332, and relationships 334) may also comprise data that represents one or more expressions that can be used to control how the corresponding ontological objects are described in narrative text produced by the narrative generation system.
For example, the entity type 322 can be tied to one or more expressions 328. When the narrative generation process determines that the subject entity type needs to be described in narrative text, the system can access the expression(s) 328 associated with the subject entity type to determine how that entity type will be expressed in the narrative text. The expression(s) 328 can be a generic expression for the entity type 322 (e.g., the name 326 for the entity type, such as the name “salesperson” for a salesperson entity type), but it should be understood that the expression(s) 32 may also or alternatively include alternate generic names (e.g., “sales associate”) and specific expressions. By way of example, a specific expression for the salesperson entity type might be the name of a salesperson. Thus, a narrative text that describes how well a specific salesperson performed can identify the salesperson by his or her name rather than the more general “salesperson”. To accomplish this, the expression 328 for the salesperson can be specified indirectly via a reference to a data field in a data set (e.g., if the data set comprises a table that lists sales data for various sales people, the expression 328 can identify a column in the table that identifies each salesperson's name). The expression(s) 328 can also define how the subject entity type will be expressed when referring to the subject entity type as a singular noun, as a plural noun, and as a pronoun.
The expression(s) 346 for the direct attributes 330a and computed value attributes 330b can take a similar form as and operate in a manner similar to the expression(s) for the entity types 322; likewise for the expression(s) 362 tied to characterizations 332 (although it is expected that the expressions 362 will often include adjectives and/or adverbs in order to better express the characterization 332 corresponding to the subject entity type 322). The expression(s) 352 for relationships 334 can describe the nature of the relationship between the related entity types so that this relationship can be accurately expressed in narrative text if necessary. The expressions 352 can typically take forms such as “within” (e.g., a “city” entity type within a “state” entity type, “belongs to” (e.g., a “house” entity type that belongs to a “person” entity type, “is employed by” (a “salesperson” entity type who is employed by a “company” entity type), etc.
Another ontological object can be a timeframe 344. In the example of
At step 372, the processor attempts to match the received text string to a base communication goal statement that is a member of a base communication goal statement library 504 (see
At step 376, the system continues to receive text string entry from the user. At step 378, the processor attempts to match the text string entry to an object in ontology 320. Is there is a match (or multiple matches), the system can present a list of matching ontological objects for user selection (step 380). In this fashion, the system can guide the user to define parameters for the communication goal statement in terms of objects known within ontology 320. However, if the text string does not match any ontological objects, the system can provide the user with an ability to create a new object for inclusion in the ontology (steps 382-384). At step 382, the system provides the user with one or more UIs through which the user creates object(s) for inclusion in ontology 320 (e.g., defining an entity type, attribute, characterization, relationship, and/or timeframe). At step 384, the system receives the user input through the UI(s) that define the ontological objects. The ontology can thus be updated at step 308 in view of the text string entered by a user that defines a parameter for the communication goal statement.
If step 310 results in a determination that the communication goal statement has not been completed, the process flow returns to step 376 as the user continues entering text. Otherwise, the process flow concludes after step 310 if the communication goal statement has been fully parameterized (see
Through the use of composable communication goal statements and ontology 320, example embodiments are capable of generating a robust array of narrative stories about data sets that satisfy flexibly-defined communication goals without requiring a user to directly author any program code. That is, a user need not have any knowledge of programming languages and does not need to write any executable code (such as source code) in order to control how the narrative generation platform automatically generates narrative stories about data sets. To the extent that any program code is manipulated as a result of the user's actions, such manipulation is done indirectly as a result of the user's higher level compositions and selections through a front end presentation layer that are distinct from authoring or directly editing program code. Communication goal statements can be composed via an interface that presents them in natural language as disclosed herein, and ontologies can similarly be created using intuitive user interfaces that do not require direct code writing.
The inventors also disclose that the ontology 320 can be re-used and shared to generate narrative stories for a wide array of users. For example, an ontology 320 can be built that supports generation of narrative stories about the performance of retail businesses. This ontology can be re-used and shared with multiple users (e.g., users who may have a need to generate performance reports for different retail businesses). Accordingly, as ontologies 320 are created for different domains, the inventors envision that technical value exists in maintaining a library of ontologies 320 that can be selectively used, re-used, and shared by multiple parties across several domains to support robust narrative story generation in accordance with user-defined communication goals.
Example Narrative Generation Architecture Using Composed Communication Goal Statements:
The platform can also include a front end presentation layer 570 through which user inputs 572 are received to define the composed communication goal statement 390. This presentation layer 570 can be configured to allow user composition of the communication goal statement 390 using natural language inputs. As mentioned herein, it can also employ structured menus and/or drag/drop features for selecting elements of a communication goal statement. Examples of various user interfaces that can be used by the presentation layer 570 are shown in
The “what to say” AI 502 can be comprised of computer-executable code resident on a non-transitory computer-readable storage medium such as computer memory. The computer memory may be distributed across multiple memory devices. One or more processors execute the computer code in cooperation with the computer memory. AI 502 operates on a composed communication goal statement 390 and ontology 320 to generate a computed story outline 528.
AI 502 includes a communication goal statement interpreter 506, which is configured to process and interpret the communication goal statement 390 to select a set of narrative analytics that are to be used to analyze a data set about which the narrative story will be generated. The computer memory may include a library 508 of narrative analytics 510 (e.g., 5101, 5102, 5103, . . . ). The narrative analytics 510 may take the form of parameterized computer code that performs analytical operations on the data set in order to facilitate a determination as to what content should be included in the narrative story so that the communication goal(s) corresponding to the communication goal statement 390 are satisfied. Examples of narrative analytics 510 can be the computational logic 392, 394, and 396 shown in
AI 502 can maintain a mapping that associates the various operators that may be present in communication goal statements (e.g., “Present”, “Compare”, etc.) to a sequence or set of narrative analytics that are to be performed on data in order to support the data analysis needed by the platform to generate narrative stories that satisfy the communication goal statement 390. Thus, the “Compare” operator can be associated with a set of narrative analytics that do simple difference (a-b), absolute difference (abs(a-b)), or percent difference ((b-a)/b). In an example embodiment, the mapping can also be based on the parameters that are included in the communication goal statement 390. The mapping can take the form of a data structure (such as a table) that associates operators (and possibly also parameters) with sets of narrative analytics 510 from library 508. Interpreter 506 can then read and interpret the communication goal statement 390 to identify the operator included in the communication goal statement, access the mapping data structure to map the identified operator to its corresponding set of narrative analytics 510, and select the mapped narrative analytics. These selected narrative analytics 512 in turn drive downstream operations in AI 502.
AI 502 can also include computer code 516 that is configured to determine the data requirements that are needed by system to generate a narrative story in view of the selected narrative analytics 512 and the parameters that are included in the communication goal statement 390. This code 516 can walk through the selected narrative analytics 512, the communication goal statement 390, and ontology 320 to identify any parameters and data values that are needed during execution of the selected narrative analytics 512. For example, the communication goal statement 390 may include parameters that recite a characterization of an entity. Computer code 390 can identify this characterization in the communication goal statement and access the ontology 320 to identify the data needed to evaluate the characterization of the subject entity such as the attribute(s) 330 and value(s) 368 needed for the subject characterization 332 in ontology 320. The ontology 320 can then be further parsed to determine the data requirements for the subject attribute(s) needed by the subject characterization 332, and so on until all data requirements for the communication goal statement 390 and selected narrative analytics 512 are determined. This ultimately yields a set of data requirements 518 that define the data needed by AI 502 in order to support the data analysis used to determine the content to be expressed in the narrative story. In situations where the input to AI 502 comprises multiple communication goal statements 390 in a story outline, code 516 can be configured to walk through the outline to assemble a list of the data requirements for all of the communication goal statements in the outline.
Once the data requirements 518 have been determined, the AI 502 can execute computer code 522 that maps those data requirements 522 to source data 540. (This can be done either in a “batch” model wherein all the data requirements are determined first, and the code to map those to source data is executed; or it can be done individually for each data requirement either as needed or as the other information necessary to make the determination becomes available.) The source data 540 serves as the data set from which the narrative story will be generated. Source data 540 can take the form of data in a database, data in spreadsheet files, or other structured data accessible to AI 502. Computer code 522 can use a data structure 520 (such as a table) that associates parameters from the data requirements to parameters in the source data to perform this mapping. For example, consider a scenario where the communication goal statement is “Present the Sales of the Salesperson”. The data requirements 518 for this communication goal statement may include a parameter that corresponds to the “sales” attribute of a salesperson. The source data 540 may include a data table where a column labeled as “Amount Sold ($)” identifies the sales amount for each salesperson in a company. The parameter mapping data structure 520 can associate the “Sales” parameter from the data requirements 518 to the “Amount Sold ($)” column in the source data 540 so that AI 502 accesses the proper data. This parameter mapping data structure 520 can be defined by an author when setting up the system, as discussed hereinafter. The output of computer code 522 can be a set of mapped source data 524 for use by the selected narrative analytics 512.
Computer code 522 can also map data requirements to source data using story variable(s) 542. For example, the communication goal statement 390 might be “Compare the Sales of Salesperson “John Smith” to the Benchmark of the Salesperson”. The mapped source data 524 can identify where in the source data the sales and benchmark for salespeople can be found. If the source data 540 includes sales data for multiple salespeople (e.g., rows in a data table correspond to different sales people while columns in the data table correspond to sales amounts and benchmarks for salespeople), the selection of a particular salesperson can be left as a story variable 542 such that the parameter mapping data structure 520 does not identify which specific row to use as the salesperson and instead identifies the salesperson data requirement as a story variable. When a user composes the communication goal statement such that “John Smith” is expressed in the statement where the salesperson parameter is located, the computer code 522 can use “John Smith” in the communication goal statement 390 as the story variable 542 that governs the selection of which row of source data 540 should be used. Similarly, the benchmark parameter might be expressed as a story variable 542. For example, the source data 540 may not include a benchmark field, but the composed communication goal statement might express a number to be used as the benchmark. In such a situation, this number could be a story variable 542 used by the system.
AI 502 can also include computer code 526 that executes the selected narrative analytics 512 using the mapped source data 524 (and potentially any story variable(s) 542) to produce a computed story outline 528. The narrative analytics 512 specifies at least four components: the input parameters (e.g., an entity to be ranked, a metric it is to be ranked by, and a group in which it is to be ranked); the code that will execute the narrative analytics (i.e., that will determine the rank of the entity in the group according to the metric); the output parameters (i.e., the rank of the entity); and a statement form containing the appropriate input and output parameters that will form the appropriate statement for inclusion in the computed outline (in this case, rank(entity, metric, group, rankvalue)). The communication goal statement 390 can be associated with a general story outline that provides the basic structure for the narrative story to be generated. However, this general story outline will not be populated with any specific data—only general identifications of parameters. Through execution of the selected narrative analytics by computer code 526, this general story outline can be populated with specific data in the form of the computed story outline 528. For example, continuing with an example from above where the communication goal statement 390 is “Compare the Sales of Salesperson “John Smith” to the Benchmark of the Salesperson”, the selected narrative analytics may include parameterized code that computes data indicative of the difference between John Smith's sales amount and the benchmark in both absolute terms (e.g., performing a subtraction between the sales amount and the benchmark) and as a percentage (e.g., dividing the subtracted difference by the benchmark and multiplying by 100). Code 526 executes these narrative analytics to compute data values for use in the story outline. These data values are then embedded as values for the parameters in the appropriate statement forms associated with the narrative analytics to produce statements for inclusion in the computed outline. The statement will be included in the computed outline as a new element of the section containing the communication goal for which it was computed, under the node representing that communication goal. Code 526 will progress through the execution of the selected narrative analytics using mapped source data 524 and story variable(s) 542 (if any) until all elements of the story outline have been populated with statements. Also associated with communication goals are characterizations that serve to express a characterization or editorialization of the facts reported in the statements in a manner that may have more narrative impact that just a reporting of the facts themselves. For example, rather than saying that an entity is ranked first, we might say that it is the best. (In another approach, these might be associated with sections rather than communication goals.) The characterizations associated with each communication goal are assessed with respect to the statements generated by the narrative analytics in response to that goal. This results in generating additional propositions or statements corresponding to those characterizations for inclusion in the computed outline in those cases when the conditions for those characterizations are met by the input statements. The characterizations are also linked to the statements which they characterize. The result of this process is a computed story outline 528 that serves to identify the content that is to be expressed in the narrative story.
The “how to say it” AI 504 can be comprised of computer-executable code resident on a non-transitory computer-readable storage medium such as computer memory. The computer memory may be distributed across multiple memory devices. One or more processors execute the computer code in cooperation with the computer memory. AI 504 employs NLG logic 530 to generate a narrative story 550 from the computed story outline 528 and ontology 320. As indicated above, objects in ontology 320 can be associated with expressions (e.g., expressions 328, 346, 352, 358, and 362) that can be used by NLG 530 to facilitate decision-making regarding the appropriate manner of expressing the content in the computed story outline 528. Thus, NLG 530 can access the ontology 320 when forming sentences from the computed story outline 528 for use in the narrative story 550. Example embodiments of NLG 530 are discussed below with reference to
Once again, by leveraging predefined sets of parameterized narrative analytics 510, AI 502 is able to shield the low level program coding from users so that a user need only focus on composing communication goal statements 390 in a natural language in order to determine the content that is to be included in a narrative story. Further still, AI 504 also operates transparently to users so that a narrative story 550 can be generated from a composed communication goal statement 390 without requiring the user to directly write or edit program code.
Example Platform Operation:
During setup, the system loads the story configuration from a configuration store. The configuration store is a database where configurations are maintained in persistent form, managed, and versioned. The configuration for a story includes items representing the outline (sections, communication goals, and their components), the ontology (entity types, relationships, timeframe types), and data connectors (sources, data mappings). Once the configuration for the story is loaded into memory, the story outline is constructed, as shown in
Once the setup phase is complete, the outline can be used to govern the generation of a story. This is accomplished by traversing the outline and executing the analytics associated with each communication goal statement; and the results serve to parameterize the associated statement forms of the communication goal in order to generate the facts of the story (see
When this generation process is invoked by a client, e.g., via an API request, the client provides certain values for parameters of the configuration. In this instance, for example, the story is about the sales of some particular salesperson. So the client may need to provide a unique identifier for the specific salesperson which can be interpreted via the mapping provided between parameters of the story outline and the data source to be used.
As shown by
Continuing with the example, the single communication goal statement in this case, “Present the Sales of the Salesperson”, is made up of two base communication goal statements, composed together by embedding one inside the other. The top level statement is AttributeOfEntity(AttributeName, <Entity>), and its Entity parameter is satisfied by the embedded statement EntityById(Id). EntityById is resolved first. This is computed by retrieving the entity's ID as provided by the client when invoking the generation process, e.g., via an API request. EntityById creates an (internal) Entity object corresponding to the (external) ID and returns that Entity object as its result. This internal Entity object is a new Entity of the appropriate Entity Type as specified in the configuration and with appropriate attributes as determined by the entity data mapping, in this instance, since we are talking about a Salesperson, relevant attributes of the Salesperson in question such as his or her name, gender, sales, office—whatever in fact the configuration specifies be retrieved or computed. This result is in the form of the embedded communication goal statement, namely, EntityById(Id, <Entity>); it is then, in turn, passed into the top-level AttributeOfEntny statement along with the attribute name “sales”. The AttributeOfEntny analytic comprises code that takes the entity object and returns the corresponding value for that attribute of the entity as its result. The analytic looks up where to get the attribute data based on the entity data mappings provided during configuration, and retrieves the specific relevant attribute data from the client's data. The results for both of these are wrapped up in statement forms to produce statements as described above, and these statements are then added to the Computed Outline. In this specific case, as mentioned above, the statements are composed by one being embedded inside the other. The resulting compound statement added to the Computed Outline in this instance, fully parameterized, would look something as follows: AttributeOfEntny(‘Sales’, EntityByID(‘1234’, Salesperson1234), 15000).
As shown by
The first phase, Model Generation, converts the compound statements in the computed outline into NLGModel graphs, as shown by
For example, the statement for AttributeOfEntny(‘Sales’, EntnyByID(‘1234’, Salesperson1234), 15000) is converted into a model graph where the root is an EntityModel representing the Salesperson1234. The EntityModel has a dependent AttributeModel representing the Sales attribute since Sales is an attribute of that entity. The attribute Sales has a value of 15000 so a ValueModel representing 15000 is added as a dependent to the AttributeModel. Finally, the ValueModel has a UnitModel representing the type of value. In this case it is ‘dollars’. This model graph now provides the structure needed for the NLG system to construct a sentence for this statement. This was a simple example. The more complicated the statement, the more complicated the model graph will be. The system can also combine multiple statements into a single big model graph assuming they are related somehow, for example each of them are about the same entity. This then allows the system to then express multiple sets of ideas in a single sentence. If the model graph is too big, i.e. there are too many ideas to express in one sentence, it is split up into reasonably sized subgraphs that make up individual sentences.
After a model graph has been generated for each node, adjacent nodes are compared with each other to mute redundant facts. This can be referred to as Model Muting, as shown by
Next, sentences are generated based on each model graph during Sentence Generation as shown by
Continuing with the working example, only the Attribute model can generate sentences for this model graph. It will generate them based on the attribute expressions configured by the user for “sales”. Let's suppose the user configured three options: “the salesperson had sales of $100”, “the salesperson sells $100”, and “the salesperson's sales are $100”. The Attribute model would generate three sentences, one for each of these options.
After the base sentences have been generated, the models not expressed in that base sentence are then be expressed as clauses on the sentence. This can be referred to as Clause Placement (see
In our example, there are no extra models that need to be added as clauses. However, to illustrate how the clause placement phase would work, let's say that the goal was actually “Present the sales of the salesperson working in the city.” A sentence from the Relationship model would be “Sally sells in Chicago.” This leaves the Attribute/Value/Unit models still needing to be expressed. The Attribute model can produce clauses for these. Based on the attribute expression configuration, it would generate clauses of “who has sales of $1000” or “who has sold $1000”. These would be added as a relative clause to “Sally” giving a complete sentence of “Sally, who has sales of $1000, sells in Chicago” (as one of the sentences among the several available permutations).
The next phase is Sentence Selection (see
At this point, the system seeks to improve fluidity by looking across the nodes in the outline. At this stage, referred to as Entity Referencing (see
In our example, since there is only a single goal there would be no pronoun replacement. If instead there were two adjacent goals in the same section (e.g., “Present the sales of the salesperson” and “Present the title of the salesperson”, a pronoun would be used for the second sentence, resulting in the language “Sally had sales of $10000. She had the title VP of Sales.”
At this point, the sentences have been finalized. The next thing to do is ensure that the sentences are grammatically correct. This phase can be referred to as Realization (see
Wrapping up the example, the realized sentence ends up being “Sally has sales of $10,000.” To get to that, the verb “has” was conjugated into present tense because the lack of a timeframe. The system can be configured to assume the timeframe is “now” in cases where no timeframe is specified in the communication goal statement. Also, the Realization phase inspects “sales” and determines that it was plural so an indefinite article was not needed. Finally, “Sally” is determined to be a name proper noun, which accordingly means that a definite article is not needed before “Sally”.
As a last step, which can be referred to as Document Generation (see
Ontology Building:
If a new attribute is to be created for the value, the process flow proceeds to step 902. At step 902, the process flow makes a decision as to whether the new attribute should be a direct attribute or a computed value attribute.
If a direct attribute is to be created, the process flow proceeds to step 904. At step 904, the processor defines a label for the attribute in response to user input. This label can serve as the name for the attribute (e.g., “sales”—see
Next, at step 908, the processor defines the expression(s) that are to be associated with the subject attribute. Through specification of one or more expressions for the subject attribute, the user can provide the system with a number of options for expressing the attribute in words when rendering a narrative story.
At step 910, the processor selects the entity type for the subject attribute in response to user input.
If step 902 results in a determination that a computed value attribute is to be created, the process flow proceeds to step 912 from step 902. At step 912, the system presents the user with a choice of making the computed value attribute a function or an aggregation (step 912). If a function is selected at step 912, the process flow proceeds to step 914 where the processor sets the computed value attribute according to the user-selected function. If an aggregation is selected at step 912, the process flow proceeds to step 916 where the processor sets the computed value attribute according to the user-selected aggregation. Examples of available aggregations can include count, max, mean, median, min, range, and total. These aggregations can be associated with corresponding parameterized computational logic (see
After the attribute has been defined via the process flow of
It should be understood that additional operations can be included in the attribute definition process flow if desired by a practitioner. For example, if a practitioner wishes to attach timeframe details to attributes, a timeframe definition process flow can be added to the
At step 1002, the system determines whether the user wants to create a new characterization or select an existing characterization. This step can be performed in a manner similarly to step 900 in
At step 1004, the user selects the attribute(s) for use in the characterization. If the attribute needs to be defined, the process flow of
At step 1006, the user sets the qualification(s) by which to evaluate the characterization. For example, these qualifications can be a series of thresholds by which the values of the sales attribute are judged (e.g., the characterization changes based on whether the sales amount are above or below a threshold of $10,000). Multiple thresholds can be defined for a characterization, which would then yield more than two potential outcomes of a characterization (e.g., three or more tiers of characterization outcomes). Also, the qualifications need not be defined in terms of fixed thresholds. The thresholds can also be flexibly defined in terms of direct attributes and/or computed value attributes (for example, a salesperson can be characterized as a satisfactory salesperson if the sales attribute for the subject salesperson has a value that exceeds the value of the benchmark attribute for the subject salesperson; as another example, a salesperson can be characterized as an above-average salesperson if the sales attribute for the subject salesperson has a value that exceeds the average value of the sales attributes for the all of the salespeople within a company). As part of defining the qualifications, step 1006 can also involve the user specifying the operators by which to judge qualifications. Examples of operators may include “greater than”, “less than”, “greater than or equal to”, “equals”, etc.
At step 1008, the user sets the expression(s) for the subject characterization. These expressions can then be used by the NLG process when articulating the subject characterization in a narrative story. For example, in a characterization relating to the performance of a salesperson in terms of sales, expressions such as “star performer”, “outperformed”, “high performer” etc. can be used in situations where the sales exceeded the highest threshold, while expressions such as “laggard”, “poor performer”, “struggled”, etc. can be used in situations where the sales were below the lowest threshold.
At step 1100, the processor decides, in response to user input, whether to create a new entity type or select an existing entity type. This step can be performed while a user is composing a communication goal statement. If step 1100 results in a determination that an existing entity type is to be used, the process flow can proceed to step 1150 where an existing entity type is selected.
If step 1100 results in a determination that a new entity type is to be created, the process flow proceeds to step 1102. At step 1102, the user provides a label for the entity type. This label can be used as the entity type's name (e.g., a “salesperson” entity type). Next, at step 1104, the user sets a base type for the subject entity type. Examples of available base types to choose from can include person, place, thing, and event. However, it should be understood that more, fewer, and/or different base types can be used. The specified base type can be used by the AI logic to inform decision-making about the types of pronouns that can be used to express the subject entity type, among other expressive qualities for the entity type.
At step 1106, the user sets one or more expressions in relation to the subject entity type. These expressions provide the NLG process with a variety of options for expressing the entity type in a story.
The
For example, a relationship can be added to the subject entity type at steps 1108-1116. At step 1110, the user identifies the entity type to which the subject entity type is to be related. If the relating entity type does not exist, the process flow of
Another example of a feature that can be added to an entity type is a rank. Steps 1120-1124 describe how a rank can be attached to an entity type. The rank feature provides the AI with a mechanism for notionally identifying entities to be discussed in a narrative story even if the user does not know in advance which specific entities are to be discussed. For example, a user may want the system to generate a story about the 3 top ranked salespeople in terms of sales, but does not know a priori who these salespeople are. The rank feature attached to the salesperson entity type allows for a user to easily compose a communication goal statement that can be used by the AI to generate an appropriate narrative story. At step 1122, the user sets the attribute by which the subject entity type is to be ranked. For example, if salespeople are to be ranked by sales, the user can specify the sales attribute at step 1122. The
Another example of a feature that can be added to an entity type is a qualification. Steps 1130-1134 describe how a qualification can be attached to an entity type. Similarly to the rank feature, the qualification feature provides the AI with a mechanism for notionally identifying entities to be discussed in a narrative story even if the user does not know in advance which specific entities are to be discussed. For example, a user may want the system to generate a story about the salespeople who have 10 years of more of experience or who have been characterized as star performers in terms of sales, but does not know a priori who these salespeople are. The qualification feature attached to the salesperson entity type allows for a user to easily compose a communication goal statement that can be used by the AI to generate an appropriate narrative story. At step 1132, the user sets the attribute 330 and/or characterization 332 that will be used to filter/qualify the subject entity type. For example, if the user wants the story to focus on salespeople with at least 10 years of experience, the user can specify a “years worked” or “start date” attribute at step 1132. The
At step 1200, the processor decides, in response to user input, whether to create a new timeframe or select an existing timeframe. This step can be performed while a user is composing a communication goal statement. If step 1200 results in a determination that an existing timeframe is to be used, the process flow can proceed to step 1212 where an existing timeframe is selected.
If step 1200 results in a determination that a new timeframe is to be created, the process flow proceeds to step 1202. At step 1202, the system determines whether the user wants to create a new timeframe type or select from among existing timeframe types. Examples of timeframe types include years, months, days, hours, etc.
If a new timeframe type is desired, the process flow proceeds to step 1204 where the user defines the timeframe type and step 1206 where the user sets the expression(s) for the timeframe type. The expression(s) provide the NLG process with a variety of options for expressing the timeframe in a story.
If an existing timeframe type is desired, the process flow proceeds to step 1208 where the user makes a selection from among existing timeframe types and step 1210 where the user defines a designation for the selected timeframe type. Through this designation, the user can define qualifications via a “when” statement or the like that defines time-based conditions (e.g., “the month of the year when the sales of the store were highest”).
As explained above, the ontology 320 defined via the process flows of
Subgoals within Communication Goal Statements:
The communication goal statements may be interpreted by the system to include a plurality of subgoals or related goals. Thus, in order for a narrative story to satisfy the communication goal associated with a communication goal statement, it may be desirable to the narrative story to first satisfy one or more subgoals related to the communication goal of the communication goal statement. An example of this is shown by
During the composition process, a user may parameterize the base communication goal statement 406 of
During the process of composing communication goal statements for use in the narrative generation process, the system can provide GUI screens to a user that allows the user to expand a communication goal statement to show communication goal statements associated with subgoals. Furthermore, the GUI can be configured to respond to user input to selectively opt in and opt out of which subgoals are to be included in the narrative generation process for a section of the story outline. Thus, if a user wants the story to include a headline or a title that is drawn from the “Compare” communication goal statement, a user can use a GUI to expand the “Compare” communication goal statement into statements for its constituent subgoals. For the headline/title, a user can choose to selectively opt out of the first two “Present” statements but retain the “Characterize” statement so that the headline/title is focused on a desired main point. Then, in the body of the narrative story, the user can selectively retain all of the constituent subgoals for the “Compare” statement so that the body of the narrative story provides the context for the comparison.
Example Embodiments for a Conditional Outcome Framework to Determine Narrative Content:
In another example embodiment, the system can employ a conditional outcome framework to support narrative generation. For example, AI 502 can employ a conditional outcome framework to determine content for inclusion in a narrative.
It should be understood that the outcome data structures 1502 can be tied together in numerous arrangements to define branching logic for the conditional outcome framework 1500. For example, there can be multiple layers of outcome data structures 1502 (each with associated conditions) to provide branching operations at multiple levels. Such branching structures allow for the conditional outcome framework 1500 to accommodate highly complex and intelligent decision-making as to what ideas should be expressed in a narrative in view of the nature of the data under consideration. Moreover, the outcome data structures 1502, idea data structures 1504, and supporting analytics 1506 can be parameterized to allow their re-use in a wide variety of contexts.
It should also be understood that the same idea data structure 1504 might be linked to multiple different outcome data structures 1502. Furthermore, a given outcome data structure 1502 might be linked to multiple idea data structures 1504. Examples of such arrangements are discussed below with reference to
Example Embodiments for “Analyze” Communication Goal Statements:
As mentioned above, an operator such as “Analyze” can be used to identify a communication goal statement corresponding to an analysis communication goal. An example of a base communication goal statement for an analysis communication goal that could be supported by the system is “Analyze Entity Group by Attribute”, where “Entity Group” serves as a parameter for a group of entities in the ontology 320 and “Attribute” serves as a parameter for an attribute of the specified entity group in the ontology 320. Such a base communication goal statement could be parameterized into a communication goal statement as “Analyze the Salespeople by Sales”, where the Entity Group is specified as “Salespeople” (which can be a group of entities in the ontology 320 that have the entity type of “Salesperson”), and where the Attribute is specified as “Sales” (which can be an attribute of a “Salesperson” in the ontology 320). However, it should be understood that such a base communication goal statement could be parameterized in any of a number of different ways. Further still, it should be understood that different base communication goal statements could be used to satisfy other analysis-related communication goals, some examples of which are discussed below.
The system can link a base communication goal statement of “Analyze Entity Group by Attribute” with narrative analytics 510 that are linked to a story structure that aims to provide the reader with an understanding of the distribution of a particular value across a group of entities. Accomplishing this may involve expressing a variety of quantitative ideas (the number of entities in the group, the average value within a group, the median value within a group, the entities with the highest and lowest values, etc.) and more qualitative ideas (the values are distributed normally, the values are distributed exponentially, the values demonstrate a “long-tail” distribution, one entity in particular had a much higher value than the other entities, etc.). Accordingly, if desired by a practitioner, the system can directly map such a communication goal statement to parameterized narrative analytics and a parameterized story configuration that will express these concepts. However, the use of a conditional outcome framework 1500 by the relevant narrative analytics can provide additional flexibility where the resulting narrative story structure will adapt as a function of not only the specified communication goal but also as a function of the underlying data.
To drive the assessments regarding group size, the supporting analytics 1506 for the conditional outcome framework can include group size characterization analytics 1600 for the various group size outcomes 1502. For example, the “tiny group” outcome 1502 can be associated with parameterized logic that determines whether the number of members of the group specified by the communication goal statement 390 is less than or equal to 1 (it should be understood that other thresholds could be used to define the boundary conditions for a “tiny group”). If so, the “tiny group” outcome 1502 would evaluate as true. As another example, the “decent sized group” outcome 1502 can be associated with parameterized logic that determines whether the number of members of the group specified by the communication goal statement 390 is between 2 and 50 (it should be understood that other thresholds could be used to define the boundary conditions for a “decent sized group”). If so, the “decent sized group” outcome 1502 would evaluate as true. As another example, the “large group” outcome 1502 can be associated with parameterized logic that determines whether the number of members of the group specified by the communication goal statement 390 exceeds 50 (it should be understood that other thresholds could be used to define the boundary conditions for a “large group”). If so, the “large group” outcome 1502 would evaluate as true.
To drive the assessments regarding distribution within the group, the supporting analytics 1506 for the conditional outcome framework can include group distribution characterization analytics 1602 for the various group distribution outcomes 1502. In this example, the system seeks to characterize (1) a “tiny group” as being an empty group (see the “empty” outcome 1502) or a single member group (see the “just one” outcome 1502), (2) a “decent sized group” as being a typical distribution (see “typical distribution” outcome 1502), a distribution that is clumpy at the top (see “clump at top” outcome 1502), or a flat distribution (see the “flat distribution” outcome 1502), and (3) a “large group” as being a normal distribution (see “normal distribution” outcome 1502) or a long-tail distribution (see the “long-tail distribution” outcome 1502). Each of these second level outcomes 1502 can be associated with parameterized analytics 1602 that specify the computations used for characterizing the nature of the distributions within the group. For example, the “clump at top” outcome 1502 can be associated with parameterized analytics 1602 that are configured to sort entities by a particular value, group entities with similar values, and then determine if the highest ranked entities constitute a subgroup of similar values. Any thresholds or parameters used in determining such subgroups may be built into the system, specified directly by users, or tuned automatically by the system. As another example, the “long-tail distribution” outcome 1502 can be associated with parameterized analytics 1602 that are configured to perform distribution analysis and then determine if a significant proportion of the entities contributed values well below the mean contribution. Again, any thresholds or parameters used could be built into the system, specified directly by users, or tuned automatically by the system.
In
Any ideas 1504 that are resolved based on the conditional outcome framework could then be inserted into the computed story outline 528 for use by AI 504 (together with their associated specifications in view of the underlying data) when rendering the desired narrative.
To the extent that any of the ideas 1504 need additional computed values in order to be expressed (where such values were not previously computed by analytics 1600 or 1602), the supporting analytics 1506 can further include idea support analytics 1604. For example, if the analytics 1600 and 1602 do not compute a mean value for the attribute values within the group, the idea support analytics 1604 can include parameterized logic that computes such a mean value for the underlying data.
Thus, it can be seen that the example conditional outcome framework for a communication goal statement can define a hierarchical relationship among linked outcomes and ideas together with associated supporting analytics to drive a determination as to which ideas should be expressed in a narrative about a data set, where the selection of ideas for expression in the narrative can vary as a function of the nature of the data set.
In example embodiments, the conditional outcome framework can be designed so that it does not need any input or configuration from a user other than what is used to compose the communication goal statement 390 (e.g., for the “Analyze Entity Group by Attribute” communication goal statement, the system would only need to know the specified entity group and the specified attribute). However, for other example embodiments, a practitioner might want to expose some of the parameters of the conditional outcome framework to users to allow further configurations or adjustments of the conditional outcome framework.
For example, a practitioner might want to implement the thresholds used within the conditional outcome framework as user-defined values. In the context of
As another example, a practitioner might want to provide users with a capability to enable/disable the links between outcomes 1502 and ideas 1504 in a conditional outcome framework. For example, a GUI could present a user with lists of all of the outcomes 1502 and ideas 1504 that can be tied to a communication goal statement within a conditional outcome framework. The user could then individually select which ideas 1504 are to be linked to which outcomes 1502. If desired by a practitioner, that conditional outcome framework can include default linkages that are presented in the GUI, and the user could make adjustments from there.
It should be understood that the system can also be designed to support other “analyze” communication goals. For example, another base communication goal statement that can be used by the system can be “Analyze Entity Group by Attribute 1 and Attribute 2”. Such a multi-attribute analysis goal can trigger the performance of tradeoff analysis as between the two attributes (and the expression of ideas that result from this analysis). For example, this goal may trigger analysis that results in quantitative ideas like the average values for Attribute 1, the average values for Attribute 2, the entity with the largest value for Attribute 1, etc. Assuming the system has an understanding of the relationship between Attribute 1 and Attribute 2 (for instance that “Attribute 1 is a driver of Attribute 2” or that higher values for Attribute 1 represent a positive outcome while higher values for Attribute 2 represent a negative outcome), the goal may also result in more qualitative ideas that capture intuitive understandings like “Entities that score have high values for Attribute 1 also have high values for Attribute 2”, “The entity with the highest value for Attribute 1 actually has a really low value for Attribute 2”, or “There's no correlation between values for Attribute 1 and Attribute 2 in the group”. Accordingly, it should be understood that it may be desirable for the narratives produced in response to the “Analyze Entity Group by Attribute 1 and Attribute 2” communication goal statement to express different ideas than the narratives produced in response to the “Analyze Entity Group by Attribute” communication goal statement.
Another example of a base communication goal statement for an “analyze” communication goal that can be used by the system can be “Analyze Entity Group by a Change in Attribute (Over Time)”. Such communication goal statement can trigger analysis that eventually results in quantitative ideas representing the total change in value, average change in value, the median change in value, which entity had the biggest change in values, the number of entities that had positive changes, etc. Such a goal might also produce more qualitative ideas that capture intuitive understandings such as “All members of the group had positive changes”, “About half of the group had positive changes and about half had negative changes”, or “The group as a whole had a positive change, but it was really a small group of entities that had large positive changes while the rest had smaller negative changes”. A practitioner may desire that narratives produced from this communication goal statement express different ideas than those generated from the other “analyze” communication goals discussed above.
Yet another example of a base communication goal statement for an “analyze” communication goal that can be used by the system can be “Analyze Entity Group by Characterization”. Such communication goal statement can trigger analysis that eventually results in quantitative ideas representing the count and percentage of entities with each characterization, the most common characterization, etc. Such a goal might also produce more qualitative ideas that capture intuitive understandings such as “There was a roughly even distribution of characterizations across the group”, “Every entity in the group had the same characterization”, “Almost all of the entities in the group had the same characterization”, etc. A practitioner may desire that narratives produced from this communication goal statement express different ideas than those generated from the other “analyze” communication goals discussed above.
Live Story Editing:
Another innovative feature that may be included in a narrative generation platform is an editing feature whereby a user can use a story outline comprising one or more composed communication goal statements and an ontology to generate a narrative story from source data, where the narrative story can be reviewed and edited in a manner that results in automated adjustments to the narrative generation AI. For example, an author using the system in an editing mode can cause the system to generate a test narrative story from the source data using one or more composed communication goal statements and a related ontology. The author can then review the resulting test narrative story to assess whether the story was rendered correctly and whether any edits should be made. As an example, the author may decide that a different expression for an entity would work better in the story than the expression that was chosen by the system (e.g., the author may decide that a characterization expressed as “slow growth” in the narrative story would be better expressed as “sluggish growth”). The user can directly edit the text of the narrative story using text editing techniques (e.g., selecting and deleting the word “slow” and typing in the word “sluggish” in its place). Upon detecting this edit, the system can automatically update the ontology 320 to modify the subject characterization object 332 by adding “sluggish growth” to the expression(s) 364 for that characterization (and optionally removing the “slow growth” expression).
To accomplish this, words in the resultant test narrative story can be linked with the objects from ontology 320 that these words express. Further still, sentences and clauses can be associated with the communication goal statements that they serve. In this fashion, direct edits on words, clauses, and sentences by an author on the test narrative story can be traced back to their source ontological objects and communication goal statements.
Another example of an innovative editing capability is when an author chooses to re-order the sentences or paragraphs in the test narrative story. Given that sentences and paragraphs in the test narrative story can be traced back to communication goal statements in the story outline, the act of re-ordering sentences and/or paragraphs can cause the system to automatically re-order the communication goal statements in the story outline in accordance with the editing. Thus, consider a story outline that comprises Communication Goal Statement 1 followed by Communication Goal Statement 2 followed by Communication Goal Statement 3 that produces a narrative story comprising Sentence 1 (which is linked to Communication Goal Statement 1), followed by Sentence 2 (which is linked to Communication Goal Statement 2), followed by Sentence 3 (which is linked to Communication Goal Statement 3). If the user decides that the story would read better if Sentence 2 came before Sentence 1, the user can perform this edit in the live story editing mode of the system, and this edit can cause the system to automatically adjust the story outline to comprise Communication Goal Statement 2 followed by Communication Goal Statement 1 followed by Communication Goal Statement 3.
Similarly, if a user edits the narrative story by deleting a sentence, the system can automatically adjust the story outline by deleting the communication goal statement linked to that sentence.
Through the automated changes to the ontology 320 and/or story outline, the system can be able to quickly adjust its story generation capabilities to reflect the desires of the author. Thus, during a subsequent execution of the story generation process, the system can use the updated ontology 320 and/or story outline to control the narrative generation process.
While the invention has been described above in relation to its example embodiments, various modifications may be made thereto that still fall within the invention's scope. Such modifications to the invention will be recognizable upon review of the teachings herein.
This appendix describes a user guide for an example embodiment referred to as Quill, and it is organized into the following sections:
A1: Introduction
A1 (i): What is Quill?
A1 (ii): What is NLG?
A1 (iii): How to use this Guide
A2: Getting Started
A2(i): Logging in
A2(ii): General Structure
A2(iii): Creating Projects
A3(i): Configure a Sales Performance Report
A4(i): Entity Types and Expressions
A4(iii): Characterizations
A4(iv): Attributes
A5(i): The Outline
A6(i): Getting Data Into Quill
A7(ii): Logic Trace
A7(iii): Monitoring
A8: Managing Story Versions
A8(i): Drafts and Publishing
A8(ii): Change Log
A9: Writing Stories in Production
A9(i): API
A9(ii): Scheduling
A10: Sharing and Reuse
A11: Terminology
A12: Communication Goal Families
A13: Miscellaneous
A13(i): Supported Chart Types
A13(ii): Supported Document Structures
A13(iii): Styling Rules
A13(iv): Using Multiple Data Views
A13(v): Permission Structure
The following sections can be read in combination with
A1: Introduction
A1(i): What is Quill?
Quill is an advanced natural language generation (Advanced NLG) platform that transforms structured data into narratives. It is an intelligent system that starts by understanding what the user wants to communicate and then performs the relevant analysis to highlight what is most interesting and important, identifies and accesses the required data necessary to tell the story, and then delivers the analysis in the most intuitive, personalized, easy-to-consume way possible—a narrative.
Quill is used to automate manual processes related to data analysis and reporting. Its authoring capabilities can be easily integrated into existing platforms, generating narratives to explain insights not obvious in data or visualizations alone.
A1(ii): What is NLG?
Natural Language Generation (NLG) is a subfield of artificial intelligence (AI) which produces language as output on the basis of data input. Many NLG systems are basic in that they simply translate data into text, with templated approaches that are constrained to communicate one idea per sentence, have limited variability in word choice, and are unable to perform the analytics necessary to identify what is relevant to the individual reader.
Quill is an Advanced NLG platform that does not start with the data but by the user's intent of what they want to communicate. Unlike templated approaches that simply map language onto data, Quill performs complex assessments to characterize events and identify relationships, understands what information is especially relevant, learns about certain domains and utilizes specific analytics and language patterns accordingly, and generates language with the consideration of appropriate sentence length, structure, and word variability. The result is an intelligent narrative that can be produced at significant scale and customized to an audience of one.
A1(iii): How to Use this Guide
Getting Started walks through how to log in to Quill and set up Organizations, Users, and Projects. It also provides an overview of the components of Quill.
Ontology Management is a high-level description of the conceptual elements stories in Quill are based on. This section will help you understand the building blocks of writing a story.
Configuring a Story from Scratch and Configuring a Story from a Blueprint talk through the steps of configuring a story in Quill. Jump to one of these sections if you want to learn the basics of using Quill.
Data Management contains the necessary information for setting up data in Quill, discussing the accepted formats and connections.
Reviewing Your Story discusses the tools available to review, edit, and monitor the stories you configure in Quill.
Managing Story Versions covers publishing stories and tracking changes made to projects.
Writing Stories in Production addresses administrative aspects of story generation, including setting up an API endpoint and scheduling story runs.
Sharing and Reuse goes through how to make components of a particular project available across projects.
Common Troubleshooting offers simple, easy-to-follow steps for dealing with common questions that arise when working in Quill.
The Terminology will help you understand the terminology used in this manual and throughout Quill, while the Communication Goal Families describes the available communication goals and how they relate to each other.
The Miscellaneous section presents an example of a state of Quill functionality.
A2: Getting Started
A2(i): Logging in
A2(i)(a): Supported Browsers
Quill is a web-based application that supports Firefox, versions 32 ESR and up, and all versions of Chrome. Logging in will depend on whether Narrative Science is hosting the application or Quill has been installed on-premises.
A2(i)(b): Hosted On-Premises
For on-premises installations of Quill, if you are an authenticated user, go to your custom URL to access Quill. You will be taken directly to your project dashboard. If you see an authentication error, contact your site administrator to be set up with access to Quill.
A2(ii): General Structure
Quill is made up of Organizations and Projects. An Organization is the base level of access in Quill. It includes Administrators and Members and is how Projects are grouped together. Projects are where narratives are built and edited. They exist within Organizations. Users exist at all levels of Quill, at the Site, Organization, and Project levels. Access privileges can be set on a per User basis and apply differently at the Site, Organization, and Project levels. (For more detail, refer to the Permissions Structure section of the Miscellaneous section.)
(ii)(a): Creating an Organization
Creating an Organization is a Site Administrative privilege. At the time that Quill is installed, whether hosted by Narrative Science or on-premises, a Site Administrator is designated. Only a Site Administrator has the ability to create an Organization (see
Site Administrators can add users, and users can only see the Organizations of which they are members. Site Administrators have access to all Organizations with the View All Dashboards option (see
Members only see the Organizations they have access to in the Organization dropdown and can toggle between them there (see
Site Administrators can use the Organization dropdown to switch between Organizations or from the Organizations page. Each Organization will have a dashboard listing Projects and People.
A2(ii)(b): Creating Users
Only an Administrator (both Site or Organization) may create a User (see
Administrative privileges cascade through the structure of Quill. (See Permission Structure in the Miscellaneous section for more information.) That is to say, an Administrator at the Organization level has Administrative privileges at the Project level as well. The Project permissions of Members are set at the Project level.
At the Project level, a user can be an Administrator, an Editor, or a Reviewer (see
An Administrator on a Project has full access, including all aspects of Authoring, sharing, drafts and publishing, and the ability to delete the Project. An Editor has access to Authoring but cannot share, publish and create a new draft, or delete the Project. A Reviewer only has access to Live Story in Review Mode. A user's access to a Project can be edited on the People tab of the Organization dashboard.
A2(iii): Creating Projects
Both Administrators and Members can create Projects from the Organization dashboard (see
The creator of a Project is by default an Administrator. When creating a new Project, select from the list of blueprint options whether it will be an Employee History, Empty Project, Municipal Expenses, Network Analysis, or a Sales Performance report (see
This is also where you can access shared components of existing projects which members of an Organization have elected to share for reuse by other Organization members. As shown by
An Empty Project allows the user to configure a Project from the ground up, and a Sales Performance Report provides the framework to configuring a basic version of a sales performance report. A user can be added to a project by clicking the plus symbol within a project (see
You can set Project level permissions using the dropdown menu (see
You can edit permissions and remove users here as well (see
Users can also be added to Projects from the People tab of the Organization dashboard (see
Each Project includes Authoring, a Data Manager, and Admin (see
Authoring is where the narrative gets built and refined; the Data Manager is where the data for the story is configured; and Project Administration is where Monitoring, the Change Log, API documentation, Project Settings, and Scheduling are located.
A2(iii)(a): Authoring
The main view in Authoring is the Outline, as shown by
The Outline is where the narrative is built. Sections can be added to provide structure and organization to the story (see
Communication Goals are then added to a Section (see
Communication Goals are one of the main underpinnings of Quill. They are the primary building blocks a user interacts with to compose a story.
Authoring is also where Entities are managed (see
An Entity is any primary “object” which has particular Attributes. It can be set to have multiple expressions for language variation within the narrative or have Relationships to other Entities for more complex representations. All of these things comprise an Ontology.
Data Requirements are how the data that supports a story is mapped to the various story elements.
Based on the Communication Goals in the Outline, the Data Requirements tab will specify what data points it needs in order to generate a complete story (see
Live Story is a means of reviewing and editing a story generated from the Outline.
It has two modes, Review mode and Edit mode. Review mode allows the user to see a complete narrative based on specific data parameters (see
Drafts and Publishing are Quill's system of managing versions of your story (see
This is how you publish your story configurations and keep a published version as read-only in order to request stories through the API or via the Scheduler. Each Project can only have one draft and one published version at a time.
A2(iii)(b): Data Manager
The Data Manager is the interface for adding the database connections or uploading the files that drive the story (see
A2(iii)(c): Project Administration
The Project Administration features of Quill are Monitoring, the Change Log, API documentation, Project Settings, and Scheduling. They are located in the Admin section of the Project.
Monitoring allows the user to see the status (success or failure) of generated stories (see
The Change Log tracks changes made to the project (see
Quill supports on-demand story generation through synchronous API access (see
Project Settings are where you can change the name of the Project and set the project locale (see
You can set your story to run at regular intervals in Scheduling (see
A3: Configure a Story from a Blueprint
The benefit of configuring a story from a project blueprint is the ability to reuse Sections, Communication Goals, Data Views, and Ontology as a starting point. These blueprints are available in the Create Project screen as discussed in the Getting Started section.
A3(i): Configure a Sales Performance Report
Select the Performance Project Blueprint and give your project a name. You can always change this later by going to Admin>Project Settings. After the project is created, you'll be taken to Authoring and presented with an Outline that has a “Headline”, “Overview”, and “Drivers” sections with associated Communication Goals within them (see
A3(i)(a): Headline
To begin, set the Attributes in the Communication Goal in the Headline. Select “the value” (see
Create an Attribute by entering “sales” and clicking “Create sales” (see
Then specify “currency” from the list of Attribute types (see
The next step in Attribute creation is to associate the Attribute with an Entity type. Since there are no existing Entity types in this blank Project, you'll have to create one (see
Click “an entity or entity group” to bring out the Entity type creation sidebar (see
Name the Entity type “salesperson” and click to create “salesperson” (see
Set the base Entity type to Person (see
Quill will make a guess at the singular and plural expressions of the Entity type. Make corrections as necessary and click “Okay” (see
There are no designations on the Entity type you created, so click “Okay” to return to the Attribute editing sidebar (see
Once an Entity type is created, it will be available for selection throughout the project. Additional Entity expressions can be added in the Entities tab (see Ontology Management).
Next, you'll specify a Timeframe for the Attribute (see
Click “Timeframe” to create a new Timeframe (see
Choose Month (see
Click “the other value” to set another Attribute (see
Name it “benchmark” (see
Associate it with the Entity type “salesperson” and set it to be in the “month” Timeframe (see
Click on the arrow to the left of the Communication Goal in the headline section (see
The bottom related goal is the Characterization (see
Check the box to opt in to the Characterization (see
Quill has default thresholds to determine the comparative language for each outcome. Entering different values into the boxes (see
A3(ii)(b): Overview
Configure the first Communication Goal in the Overview section (see
Set the Attribute of the first “Present the value” Communication Goal to be “sales in the month of the salesperson,” and the Attribute of the second “Present the value” Communication Goal to be “benchmark in the month of the salesperson” (see
Link the two Present Communication Goals by dragging (using the gripper icon on the right side of the Communication Goal that is revealed when you hover your cursor over the Goal —see
A3(iii)(c): Drivers
Step One: Click “the value” in the first Communication Goal in the Drivers section to set the Attribute. Choose computed value in the Attribute creation sidebar and go into the functions tab in order to select “contribution” (see
Set the Attribute to be “sales” (see
Click the first entity and create the new Entity type “sector” of type “Thing” (see
Add a relationship (see
Set the relationship as “managed by” (see
Add a group analysis and set the Attribute as “sales” and the Timeframe to “month” (see
Set the second entity to “salesperson” and the timeframe to “month” (see
Step Two: Follow the steps as above to complete the second Communication Goal in the Drivers section but set the position from top to be 2 in the group analysis (see
Step Three: Click into the “Search for a new goal” box and select “Call out the entity” (see
Set the entity to be highest ranking sector by sales in the month managed by the “salesperson” (see
Then move the goal by grabbing the gripper icon on the right side to the first position in the section (see
Step Four: Create another Call out the entity Communication Goal (see
Create a new Entity type of “customer” and set the base entity type to “thing” (see
Add a group analysis and set the Attribute to “sales” and the Timeframe to “month” (see
Then add a relationship and set the related entity to be “highest ranking sector by sales in the month managed by the salesperson” and choose the relationship “within” (see
Then move it to the third position in the Drivers section, after the first Present goal (see
Step Five: Create another Call out the entity Communication Goal and set the entity to “second highest ranking sector by sales in the month managed by the salesperson” (see
And move it to the fourth position in the Drivers section, before the second Present goal (see
Step Six: Create another Call out the entity Communication Goal. Create a new entity type of customer following Step Four, but set the related entity to be “second highest ranking sector by sales in the month managed by the salesperson” (see
Step Seven: Finally, create another Call out the entity Goal. Create a new plural Entity type of “regions” and set its type to be “place.” Add a group analysis and set the number from top to “3,” the Attribute to “sales,” and the Timeframe to “month” (see
Then add a relationship, setting the related Entity type as “salesperson” and the relationship as “managed by” (see
The completed outline should match
A3(iv)(d): Adding Data
In order to complete the Data Requirements for the story, you add a Data Source to the Project. Go the Data Manager section of the Project to add a Data View (see
Choose to Upload a file and name the Data View (see
Once Quill has saved the Data View to the Project, you will be presented with the first few rows of the data (see
A3(v)(e): Data Requirements
The Data Requirements will guide you through a series of questions to fill out the necessary parameters for Narrative Analytics and Communication Goals (see
See the Data Requirements section of Configure a Story from Scratch for more detail. The completed Data Requirements can appear as shown by
Go to Live Story to see the story (see
Toggles for “salesperson” (see
A4: Ontology Management
A4(i): Entity Types and Expressions
Entity types are how Quill knows what to talk about in a Communication Goal. An Entity type is any primary “object” which has particular Attributes. An example is that a Department (entity type) has Expenses (Attribute)—see
In other words, if you have an Entity type of Department, Quill will express a specific instance of a Department from your data, such as Transportation. Likewise, Expenses will be replaced with the numerical value in your data. Quill also allows you to create Entity and Attribute designations, such as departments managed by the top salesperson or total expenses for the department of transportation (see
When you generate a story with such designations, Quill replaces them with the appropriate calculated values.
A4(i)(a): Entities Tab
Entity types are managed in the Entities tab (see
Quill defaults to showing all Entity types, but you can filter to only those that are in the story (see
Clicking an Entity type tile allows you to view its details and edit it. Here, you can modify or add Entity expressions (see
4(i)(b): Creating an Entity Type
Entity types can be created from the Entities tab (see
When you create an Entity type, you select its base Entity type from the options of Person, Place, Thing, or Event (see
This gives Quill context for how to treat the Entity. In the case of the Person base Entity type, Quill knows to determine gender and supply an appropriate pronoun.
Entity types can have multiple expressions. These are managed in the Entities tab of a project (see
They can be added either from the Entities tab (see
To add expressions, open the details for an Entity type (by clicking on “salesperson,” as shown above) and click in the text area next to the plus icon in the sidebar. Type in the expression you want associated with the Entity. You can add expressions for the Specific, Generic Singular, and Generic Plural instances of the Entity by clicking on the arrow dropdown in the sidebar to toggle between the expressions (see
Attributes can be referenced in Specific entity expressions by setting the attribute name off in brackets. For example, if you would like the last name of the salesperson as an expression, set “last name” off in brackets as shown in
You can also opt into and out of particular expressions. If you have multiple expressions associated with the Entity, Quill will alternate between them at random to add Variability to the language, but you can always uncheck the box to turn the expression off (see
A4(ii): Relationships
Entity types can be tied to each other through Relationships. For example, a City contains Departments, and Departments are within a City (see
They can also be added to an existing Entity type by editing the Entity type in Authoring.
An Entity type can support multiple relationships. For example, Department has a relationship to City: “within cities”; and a relationship to Line Items: “that recorded line items” (see
A4(ii)(a): Creating a Relationship
If the Relationships already set in Quill do not meet your needs, you can create your own. Type the relationship you want to create in the “search or create” textbox and click “Create new relationship” at the bottom of the sidebar (see
After that, you will be taken through some steps that tell Quill how the new Relationship is expressed. Enter in the present tense and past tense forms of the Relationship, and Quill automatically populates the noun phrase that describes the relationship between the Entities (see
Once you complete the steps for both directions of the relationship (see
A4(iii): Characterizations
Characterizations are editorial judgments based on thresholds that determine the language used when certain conditions are met. Characterizations can be set on Entity types directly or when comparing Attributes on an Entity in a Communication Goal.
A4(iii)(a): Entity Characterizations
An Entity characterization allows you to associate descriptive language with an Entity type based on the performance of a particular Attribute. For example, you might want to characterize a Sales Person by her total sales (see
Click “+ Characterization” to create a Characterization (see
Once you've named and created the Characterization, you'll have to set the expressions for the Default outcome. Click the grey parts of speech to edit the expression in the sidebar (see
To add an Outcome, click “+ Outcome” (see
Change the Outcome label to describe the outcome. For this example, the Outcome label will be “Star” to reflect an exceptional sales performance. Again, edit the expressions by clicking on the grey parts of speech. In order for the outcome to be triggered under specific conditions, you need to add a Qualification (see
Click “+ Qualification” to set the value to Sales (see
You have a choice for comparing the value to an Attribute or a static value (see
In this case, choose to keep it a static value and set the value to $10,000 (see
Follow the same steps to create the lower bound outcome, setting the label as “laggard” and the static value to $1,000 (see
Once you have defined Characterizations on an Entity, you can include them in your story by using the Present the Characterization of the entity Communication Goal (see
A4(iii)(b): Assessment Characterizations
To set the characterizations on a comparative Communication Goal, expand the arrow to the left of the Communication Goal (see
This exposes the list of available subgoals (see section below). At the bottom of this list is a goal to assess the difference between the attributes. Check the box to expose the thresholds applied to the comparison (see
Quill has default thresholds to determine the comparative language for each outcome. These thresholds can be changed by entering different values into the boxes. If a value is changed to be less than the upper bound or greater than the lower bound of a different outcome, Quill will adjust the values so that there is no overlap (see
There is also default language to correspond with each of the possible outcomes. This can also be changed to suit your particular needs and the tone of your story. Click on the green, underlined text to open a sidebar to the right where you can add additional expressions and set which expression you would like to be the primary characterization (see
You can also opt into and out of particular expressions. However, in the example of Appendix A, you cannot opt out of whichever expression is set as the primary characterization. If you have multiple expressions associated with the outcome (see
These expressions can also be edited in Edit mode in Live Story (see
A4(iv): Attributes
An Attribute is a data-driven feature on an Entity type. As described above, Quill will express a specified Attribute with the corresponding value in the data based on your Communication Goal. Quill also supports adding modifiers to attributes in order to perform calculations on the raw value in the data.
A4(iv)(a): Attribute Values
Attribute Values are those values that are taken directly from your data. In other words, no computations are performed on them. An example is the Name of the City. If there is a value in the data for the total expenses of the city, Quill pulls this value directly and performs no computations, unless a data validation rule is applied e.g. “If null, replace with Static Value.” which is set in the Data Requirements when mapping the Outline's information needs to your Data View.
You also have the option of specifying a Timeframe (see
This allows you to restrict the window of analysis to a particular day, month, or year.
Create a new Timeframe by selecting one of those three options. Once you've done this, Quill also recognizes the “previous” and “next” instances of that Timeframe (see
A4(iv)(b): Computed Attributes
On the other hand, if the total expenses of the city are calculated by taking the sum of the expenses for each department, Quill allows you to create a Computed Value. Computed Values allow you to compute new values from values in your data and use them for group analysis.
Computed Values can be aggregations or functions. Aggregations include count, max, mean, median, min, range, total (see
In the example of Appendix A, current functions are limited to contribution, which evaluates how much of an aggregate a component contributed (see
Computed Values can be created from Present or Callout Communication Goals. When you create the attribute you are presenting or using to filter the group of Entities, click into the Computed Value tab to access the list of aggregations and functions.
A5: Configure a Story from Scratch
Quill allows you to build a story based on an existing blueprint or entirely from the ground up. To build a story specific to your needs, choose to create a Blank Project Blueprint and name it.
A5(i): The Outline
Once you've created your project, you'll be taken to the Outline (see
The Outline is a collection of building blocks that define an overall Story. This is where you do the work of building your story.
A5(i)(a): Sections
Create and name Sections to organize your story (see
Once created, a Section can be renamed, deleted, or moved around within the outline. Sections are how Communication Goals are grouped together.
A5(i)(a)(1): Renaming a Section
Click the name of the Section and type in the new name.
A5(i)(a)(2): Deleting a Section
Hover your cursor over the Section you want to delete. On the right side, two icons will appear: an ellipses and a gripper icon (see
Click the ellipses to reveal the option to delete the Section (see
If deleted the Section will disappear from the outline along with any Communication Goals it contains.
A5(i)(a)(3): Moving a Section
As above for deleting a Section, hover your cursor over the Section you want to move. Click and hold the gripper icon (see
A5(i)(b): Communication Goals
Communication Goals provide a bridge between analysis of data and the production of concepts expressed as text. In other words, they are the means of expressing your data in language.
A5(i)(b)(1): Creating a Communication Goal
Click the text box where it says to Search for a new goal. Choose the Communication Goal you′d like to use (see
A5(i)(b)(1)(A): Entity Types
Depending on the Communication Goal you choose, you will have to set the Entity type or types it is talking about. An Entity type is any primary “object” which has particular Attributes. An example is that a Department (Entity type) has Expenses (Attribute). An Entity is a specific instance of an Entity type, with data-driven values for each Attribute.
In the example of the Communication Goal “Call out the entity”, the example embodiment for Quill of Appendix A requires that an Entity type be specified. What, in your data, would you like to call out? Click “the entity” in the Communication Goal to open a sidebar to the right (see
Here you can select among Entity types that already exist or create a new one. Available entities include entities created from the outline or the entities tab (including any characterizations).
A5(i)(b)(1)(B): Creating an Entity Type
Click “new” in the Entity sidebar (see
Quill lets you know the state of an Entity type, whether it is unset, in progress, or valid based on the appearance of the Entity type in the Communication Goal. The Entity type appears grey when unset (see
Adding a relationship allows you to tell Quill that an Entity is related to another Entity. To do so, choose to Add Relationship as you create your Entity type. Then set or create the Entity type that this Entity has a relationship to (see
When creating an Entity type of the base type event (see
A5(i)(b)(1)(C): Creating a Relationship
If the Relationships already set in Quill do not meet your needs, you can create your own. Type the relationship you want to create in the “search or create” textbox and click “Create new relationship” at the bottom of the sidebar (see
After that, you will be taken through some steps that tell Quill how the new Relationship is expressed. Enter in the present tense and past tense forms of the Relationship, and Quill automatically populates the noun phrase that describes the relationship between the Entities (see
Once you complete the steps for both directions of the relationship (see
You can also apply Group Analysis to an Entity type (see
In the example of Appendix A, rank is supported. This allows you to specify which Entity in a list of Entities to use in a Communication Goal. Select whether you are asking for the position from the top or the position from the bottom and the ranking of the Entity you want (see
You also have the option of specifying a Timeframe (see
This allows you to restrict the window of analysis to a particular day, month, or year (see
Create a new Timeframe by selecting one of those three options. Once you've done this, Quill also recognizes the “previous” and “next” instances of that Timeframe (see
Once you have completed the steps to create an Entity type, Quill adds it to the list of Entity types available for use throughout the story. In other words, you can use it again in other parts of the Outline.
A5(i)(b)(1)(D): Characterizations
Characterizations are editorial judgments based on thresholds that determine the language used when certain conditions are met. Characterizations can be set on Entity types directly or when comparing Attributes on an Entity in a Communication Goal.
Refer to Characterizations in Ontology Management for more information on Entity Characterizations.
To set the characterizations on a comparative Communication Goal, expand the arrow to the left of the Communication Goal (see
This exposes the list of available subgoals (see section below). At the bottom of this list is a goal to characterize the difference between the attributes. Check the box to expose the thresholds applied to the comparison (see
Quill has default thresholds to determine the comparative language for each outcome. These thresholds can be changed by entering different values into the boxes. If a value is changed to be less than the upper bound or greater than the lower bound of a different outcome, Quill will adjust the values so that there is no overlap (see
There is also default language to correspond with each of the possible outcomes. This can also be changed to suit your particular needs and the tone of your story. Click on the green, underlined text to open a sidebar to the right where you can add additional expressions and set which expression you would like to be the primary expression (see
If you have multiple expressions associated with the outcome (see
A5(i)(b)(2): Deleting a Communication Goal
To delete a Communication Goal, hover your cursor over it to reveal a trash can icon (see
A5(i)(b)(3): Moving a Communication Goal
Moving a Communication Goal is done the same way as moving a Section. Hover your cursor over the Communication Goal to reveal the gripper icon (see
Click and move the Communication Goal within the Section or to another section (see
Communication Goals without space between them are Linked Goals, described below.
A5(i)(b)(4): Linked Goals
Quill supports linking Communication Goals. This allows the user to express ideas together. For example, you may wish to talk about the number of departments in a city along with the total budget for the city. Hover your cursor over the Communication Goal to reveal the gripper icon, click and drag it above the goal you wish to link (see
When you link the Communication Goal that expresses the number of departments and the Communication Goal that expresses the total budget for the city (see
A5(i)(b)(5): Related Goals (Subgoals)
Some goals support related goals, or subgoals. This allows you to include supporting language without having to create separate Communication Goals for each related idea. For example, if you have a Communication Goal comparing attributes on an entity—in this case, the budget and expenses of the highest ranking department by expenses within the city—you may also wish to present the values of those attributes. Expand the Communication Goal to expose those related goals and opt into them as you like (see
A5(i)(b)(6): Styling Communication Goals
Quill allows for styling Communication Goals for better presentation in a story. Hover your cursor over a Communication Goal to reveal the “Txt” dropdown on the right side (see
Here, you can choose whether the language expressed is styled as a headline (see
A5(i)(b)(7): Charts
Charts are supported for two Communication Goals: Present the [attribute] of [a group] and Present the [attribute] of a [group of events]. For either of these goals, to get a chart, go to the Txt dropdown and select Chart (see
This will render the Communication Goal as a chart.
Present the [attribute] of [a group] (see
Present the [attribute] of [a group of events] (see
A5(i)(c): Data Requirements
Once you have configured your story, Quill will ask where it can find the data to support the Entity types and Attributes you have specified in the Communication Goals. Go to the Data Requirements tab in Authoring to provide this information (see
The Data Requirements will guide you through a series of questions to fill out the necessary parameters for Narrative Analytics and Communication Goals. For each question, select the data view where that data can be found and the appropriate column in the table.
A5(i)(c)(1): Tabular Data
A5(i)(c)(2): Document-Based Data
Where the value supplied is numerical, Quill will provide analytic options for cases where there are multiple values (see
For each Entity type, Quill will ask for an identifier (see
This is what Quill uses to join data views. An identifier has no validation options as it doesn't actually appear in the story. (Data Validation is discussed below.)
The final question in Data Requirements will be to identify the main Entity the story is about (see
In the city budget example, Quill needs to know what city the story will be about. This can be set as a static value (e.g. Chicago) or as a Story Variable (see
A Story Variable allows you to use a set of values to trigger stories. In other words, if your data contains city budget information for multiple cities, setting the city the story is about as a Story Variable will allow you to run multiple stories against the same dataset. The location of the value for the Story Variable is defined earlier in Data Requirements where Quill asks where to find the city.
If there is a Timeframe in the Headline of the story, Quill will need you to identify this in Data Requirements as well.
As with the entity, this can be a static value or a Story Variable. It can also be set as the run date (see
A5(i)(d): Data Formatting
Quill allows you to set the format for certain data points to have in your data source so it can be mapped to your Outline. These formats are set based on the ontology (Entities, Attributes, etc.) being used in your Communication goals, with default styling applied to values. See the Miscellaneous section for specific styling information. As you configure the appropriate data formats present in your data view, validation rules can be applied if the types do not match for a particular story run. For example, if Quill is expecting the expenses of a city to be a currency and receives a string, the user is provided with various options of actions to take. These are specified in the Data Validation section below. To select the format of any date fields you may have, go to the Data Requirements tab in Authoring and click the checkbox icon next to a date (see
Click on the date value to open a list of date format options and make your selection (see
A5(i)(e): Data Validation
Quill supports basic data validation. This functionality can be accessed in Data Requirements. Once you specify the location of the information in the data, a checkbox appears next to it. Click this to open the data validation sidebar (see
You will be presented with a number of options in a dropdown menu for what to do in the case of a null value (see
You can tell Quill to fail the story, drop the row with the null value, replace the null value with a value you provide in the text box below, or ignore the null value.
A6: Data Management
Quill allows for self-service data management. It provides everything you need to upload files and connect to databases and API endpoints.
A6(i): Getting Data Into Quill
Quill supports data in tabular or document-based formats. Tabular data can be provided to Quill as CSV files or through table selections made against SQL connections (PostgreSQL, Mysql, and Microsoft SQL Server are supported). Document-based data can be provided by uploading a JSON file, creating cypher queries against Neo4j databases, a MongoDB connection, or through an HTTP API connection (which you can also set to elect to return a CSV).
A6(i)(a): Uploading a File
You can upload a CSV or JSON file directly to Quill in the Data Manager. In the Views tab, choose to Upload a file from the Add a Data View tile (see
Provide the name of the view and upload the file. The amount of time it will take to upload a file depends on the size of the file for a maximum file size of 50 MB, and operating against a data base connection is recommended. This automatically populates the Source Name.
Quill also supports uploading multiple data sources into one Data View. This functionality can be accessed in the Data View by clicking the three dots icon (see
Here, you can upload additional files or add additional connections (see
A6(i)(b): Adding a Connection
You can also provide data to Quill by connecting to a SQL database, a cypher query against a Neo4j database, a MongoDB database, or an HTTP API endpoint. You can add a connection from the Data View tab by choosing Start from Connection from the Add a Data View tile (see
Quill will ask for the appropriate information to set up each type of connection.
The connection will be made, subject to network latency and the availability of the data source. Data Views from connections are made from the Views tab. Choose Start from a Connection and select the connection you created (see
Quill will prompt you to specify the table to add the data source. For neo4j connections, you will have to put in a cypher query to transform the data into tabular form (see
A7: Reviewing Your Story
Once you have configured your story with Sections and Communication Goals, and satisfied the Data Requirements against a data source, you can review or edit its contents, understand the logic Quill used to arrive at the story, and monitor the status of stories you run.
A7(i): Live Story
Live Story is where you can see the narrative expression of the story you configured in the Outline (see
If you have set up your story to be based on Story Variables (as opposed to a static value), you can toggle between them (see
You can also switch between data sources (see
Click the “rewrite” button to generate a new narrative to see how any additional expressions you have added affect the Variability of the story (see
Live Story has two modes: Edit and Review.
A7(i)(a): Edit Mode
Edit mode allows you to make changes to the language in your story (see
A7(i)(a)(1): Entity Expressions
You can add Entity expressions from Live Story (in addition to the Entities tab). If you click on any Entity (highlighted in blue under the cursor) (see
You can add Entity expressions by typing in the area next to the plus sign. You can also opt into and out of particular expressions. If you have multiple expressions associated with the Entity, Quill will alternate between them at random to add Variability to the language. Click the rewrite button to see how your story changes. As described in the Ontology Management section, you can also click, hold, and drag an expression to the top of the list and opt out of the additional expressions to set it as primary.
A7(i)(a)(2): Characterization Expressions
You can edit the expressions in any Characterizations you have set on Compare Communication Goals from Edit mode in Live Story. As with Entity expressions, Characterization expressions will be highlighted in blue when you move the cursor over them (see
Click on the expression to open a sidebar to the right where you can add additional expressions and set which expression you would like to be the primary expression (see
Quill will alternate between them at random to add Variability to the language. These additional expressions will be tied to the specific Communication Goal where you added them and will not appear for others. You can also opt into and out of particular expressions, as well as delete them using the x. However, you cannot opt out of whichever expression is set as the primary expression. See Assessment Characterizations in Ontology Management for more detail.
A7(i)(a)(3): Language Guidance
You can add set Language Preferences, such as word order choice, to your story in the Edit mode of Live Story using Language Guidance. Hover over a section (sections correspond to Sections in the Outline) of the story to reveal a Quill icon on the right side (see
Click it to isolate the section from the rest of the story (see
Click on a sentence to expose any additional expressions you can opt into (see
Quill generates expressions using language patterns appropriate to the Communication Goal, so the number of additional expressions will vary and not all sentences will have additional expressions. Quill will alternate between them at random to give your story more language variation.
A7(i)(b): Review Mode
Project Reviewers have access to this aspect of Authoring. In review mode (see
A7(ii): Logic Trace
Quill allows you to see the steps it takes to express Communication Goals as a story. If you click on any sentence in the story in Live Story in Review mode, Quill will show the underlying Communication Goal or Goals (see
Expand the arrow on the left of the Goal to see the steps Quill took to retrieve data based on the Communication Goal and Data Requirements (see
In this case, it created a Timeframe and an Entity Type. Then it “shows its work” of pulling the Attribute Value of “sales” constrained by the Timeframe of “month” and associated with the Entity Type “Salesperson 1.”
The Logic Trace can also be downloaded as a JSON file from the Monitoring tab in Admin (see
A7(iii): Monitoring
You can monitor the status of any stories you run, whether they were written in Live Story or generated through API requests in the Monitoring tab in Admin. Here, you can see whether stories succeeded or failed, and filter for specific stories using the available filters below (see
Use the Newer and Older buttons to scroll through the stories (see
A8: Managing Story Versions
Quill supports creating and keeping track of changes to and versions of the stories you configure.
A8(i): Drafts and Publishing
Once you have configured your story and are satisfied with its expression in Live Story, you can Publish the draft of your story (see
Once Published, your story will go live and that version will be the one that Quill uses when stories are requested through an API connection. After a draft has been Published, any changes you wish to make to the Project should be made after creating a new draft (see
Once a new draft has been created, it can be deleted. You can also switch to the Published version if you want to abandon the changes you have made in the new draft. The drafts and publishing dropdown is also where you can save the Project as a blueprint to share with others in the Organization (see
Project Administrators are the only ones with draft creation and publishing privileges. While Editors may make changes to active drafts, they cannot publish them or create new ones. Reviewers only have access to review mode in Live Story and cannot create, make changes to, or publish drafts.
A8(ii): Change Log
Quill tracks configuration changes made within a Project. Anytime a user makes a change or adds a new element to a Project, it's noted in the Change Log. The Change Log can be accessed in the Admin section of Quill (see
Here, you can see a list of all changes in the Project, the users that made the changes, the date and time the changes were made, and the version of the project the changes were made to. As with Monitoring, you can page through the list of changes by clicking on the Newer and Older buttons (see
The Time, User, and Version information can be used to filter the list by using the drop-downs next to the column headers.
You can also download the changes made as a CSV (see
A9: Writing Stories in Production
A9(i): API
Quill supports on-demand story generation by connecting to an API. The documentation can be accessed from Admin.
API request samples are available in the API Documentation tab of the Admin section of Authoring (see
The Request Builder allows the user to select the dataset, set the format (Plain Text, HTML, JSON, or Word) of the output, and choose the syntax of the request sample (see
An external application can use the sample to post requests to the API to generate stories from Quill once the text in red has been replaced with its specific variables (see
Each Quill user will be able to request a certificate and key from their system administrator.
A9(ii): Scheduling
Stories can also be run on a schedule (see
Once Scheduling is enabled (see
A10: Sharing and Reuse
Projects can be shared with other users. The Draft dropdown menu includes an option to Save as Blueprint (see
Here, you can give the shared version of the Project a name and description (see
You can also specify how much of the Project you make available for sharing. You can include the Outline, Ontology (Entities), and Data Sources, the Outline and Ontology, or just the Outline (see
Projects that have been saved as blueprints can be accessed when choosing a blueprint. Quill defaults to including all shared projects, but you can filter blueprints based on what elements they include (Outline, Ontology, Data Sources) (see
A11: Terminology
The following provides a glossary for various terms used in connection with describing the example embodiment of Appendix A.
An Organization is a collection of Projects managed by an Administrator. Members of an Organization have access to those Projects within it that they have permissions for. Outlines are collections of building blocks that define an overall Story.
Communication Goals provide a bridge between analysis of data and the production of concepts expressed as text.
Narrative Analytics generate the information needed by Communication Goals to generate stories.
Projects are where stories are configured. A Project includes Authoring, the Data Manager, and Admin.
Project Blueprints are templates comprised of an Outline, specific story sections, and collections of Communication Goals.
An Ontology is a collection of Entity Types and Attributes, along with their expressions, that powers how Quill expresses your story.
An Entity Type is any primary “object” which has particular Attributes. An example is that a Sales Person (entity) has Sales (attribute). Relationships provide context for entities within a story.
Every Entity Type has a Base Entity Type that identifies to Quill whether it is a Person, Place, Thing, or Event.
Computed Values are a way of reducing a list of values into a representative value. The currently available aggregations are count, maximum, mean, median, minimum, and total, and the currently available function is contribution.
Characterizations are editorial judgments based on thresholds that determine the language used in communication goals when certain conditions are met.
Expressions are the various words Quill uses to express a particular concept generated by the combination of executing Narrative Analytics and Story Elements.
A Timeframe is a unit of time used as a parameter to constrain the values included in the expression of a Communication Goal or story.
Variability is variation in the language of a story. Variability is provided through having multiple Entity and Characterization expressions as well as option into additional sentence expressions through Language Guidance.
Authoring includes the Outline, Data Requirements, and Live Story. This is where you configure Communication Goals, map Entity Types and Attributes to values in the data, and review generated stories.
Data Requirements are how a user tells Quill the method by which we will satisfy a Communication Goal's data requirements. These are what a Narrative Analytic and Communication Goal need to be able to express a concept. These are satisfied either directly by configuration of the data requirements or through the execution of Narrative Analytics.
A Story Variable is the focus of a story supplied at runtime as a value from a data source (as opposed to a static value).
A Draft is an editable version of the story in a Project. Project Administrators and Editors have the ability to make changes to Drafts. Project Administrators can publish Drafts and create new ones.
The Data Manager is the part of the Project where Data Views and Data Sources backing the story are managed. This is where files are uploaded and database connections are added.
A Data View is a used by Quill to map the Outline's information needs against Data Sources. A Project can be backed by multiple Data Views that are mapped using Identifiers in the schemas.
A Data Source is a file or table in a database used to support the Narrative Analytics and generation of a story.
Admin allows you to manage all aspects of story generation other than language and data. This is where Monitoring, the Change Log, API Documentation, Project Settings, and Scheduling are located.
A12: Communication Goal Families
The example embodiment of Appendix A supports three communication goal families: Present, Callout, and Compare.
The Present goal family is used to express an attribute of a particular entity or group of entities.
Most Present goal statements have the form “Present the attribute (or computed value) of the specified entity/group.” For example:
The two exceptions to this form are when the Count or Contribution computed values are used, in which case the statements look like this:
The Callout goal family is used to identify the entity or group of entities that has some editorially-interesting position, role, or characteristics. E.g. the highest ranked salesperson, franchises with more than $1 k in daily sales, players on the winning team, etc.
Every Callout goal statement has the same structure: “Callout the specified entity/group.” For example:
The Compare goal is used to compare the values of two attributes on the same entity. Every Compare goal has the same structure: Compare the first attribute of the specified entity to the second attribute. For example:
A13(i): Charts
Quill is able to express certain configured goals as Charts, such as Bar and Line. These have default styling and colors and are guided by the Communication Goal's Narrative Analytics. Charts are supported in each available output format.
A13(ii): Supported Document Structures
Generally, Quill supports documents that are homogenous (uniformly structured) with stable keys. Example permutations of supported structures are described below.
A13(ii)(a): Single Document
In this example, as long as all documents contain the same keys (in this case, “a”, “b”, and “c”) Quill can use this data structure.
A13(ii)(b): Nested Documents
Documents with other documents nested within them are supported, though the nested documents must be homogenous with stable keys across documents.
A first example is:
A second example is:
A13(ii)(c): Unsupported Structures
The example embodiment of Appendix A does not support heterogeneous documents (non-uniform) or documents where values are used as keys.
A13(iii): Styling Rules
Quill does not use Oxford commas. So it writes like “Mary spoke with Tom, Dick and Harry” and not like “Mary spoke with Tom, Dick, and Harry.”
Quill puts one space between sentences.
Year: Datetimes that are just years are expressed numerically.
2016→“2016”
1900→“1900”
Month and Year: Datetimes that are just months and years have written out months and numeric years.
2016-03→“March 2016”
2015-11→“November 2015”
Day, Month, and Year: Datetimes that are full dates are written out months with numeric days and years.
2016-03-25→“Mar. 25, 2016”
2015-11-05→“Nov. 5, 2015”
Percents are rounded to two places, trailing zeros are removed, and a “%” is appended.
53.2593→“53.26%”
53.003→“53%”
Ordinals are written with numerical contractions.
1→“1st”
2→“2nd”
3→“3rd”
556→“556th”
Decimals are written out with decimal parts and commas inserted.
1.1→“1.1”
1.9→“1.9”
123456789→“123,456,789”
Currencies are currently assumed to be USD. In the future, they can be locale-specific (e.g. Euros). They're styled differently based on how big they are.
Less than One Thousand
Rounds to two decimal places. There are always two decimal places.
3→“$3.00”
399.9999→“$400.00”
Less than Ten Thousand
Rounds to an integer.
5000.123→“$5,000”
4171→“$4,171”
Less than One Million
Rounds to thousands with zero decimal places, appends a “K”
500,000→“500K”
123,456.789→“123K”
Less than One Billion
Rounds to millions with one decimal place if necessary, appends an “M”
500,000,000→“500M”
500,100,000.12→“500.1M”
Less than One Trillion
Rounds to billions with two decimal places if necessary, appends an “M”
500,000,000,000→“500B”
500,100,000,000.12→“500.1B”
500,130,000,000.12→“500.13B”
The following datetime formats are supported in Quill.
01/31/15
01/31/2015
31 Jan. 2015
Jan. 31, 2015
Tuesday, Jan. 31, 2015
Tuesday, Jan. 31, 2015, 01:30 AM
2015-01-31T01:30:00-0600
20150131
2015-01-31 13:30:00
01-31-2015 01:30:45
31-01-2015 01:30:45
1/31/2015 1:30:45
01/31/2015 01:30:45 AM
31/01/2015 01:30:45
2015/01/31 01:30:45
A13(iv): Using Multiple Data Views
Users can satisfy their outline's data requirements using multiple data views. While it may often be more straightforward to create a de-normalized view in the source database, the following use cases are supported. These apply to both tabular and document-based data sources.
Quill can return the Gender from Data View 2 associated with the Sales Person's ID in Data View 1 using the Sales Person ID.
Data View 1
Data View 2
Quill can match the Transactions in Data View 2 to the Sales People in Data View 1 by Sales Person ID.
Data View 1
Data View 2
A13(v): Permission Structure
This patent application is a continuation of U.S. patent application Ser. No. 16/047,837, filed Jul. 27, 2018, entitled “Applied Artificial Intelligence Technology for Narrative Generation Based on a Conditional Outcome Framework”, now U.S. Pat. No. 10,943,069, which (1) claims priority to U.S. provisional patent application Ser. No. 62/539,832, filed Aug. 1, 2017, and entitled “Applied Artificial Intelligence Technology for Narrative Generation Based on Analysis Communication Goals”, and (2) is also a continuation-in-part of (i) U.S. patent application Ser. No. 15/897,331, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Performing Natural Language Generation (NLG) Using Composable Communication Goals and Ontologies to Generate Narrative Stories”, now U.S. Pat. No. 10,762,304, (ii) U.S. patent application Ser. No. 15/897,350, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Determining and Mapping Data Requirements for Narrative Stories to Support Natural Language Generation (NLG) Using Composable Communication Goals”, now U.S. Pat. No. 10,585,983, (iii) U.S. patent application Ser. No. 15/897,359, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Story Outline Formation Using Composable Communication Goals to Support Natural Language Generation (NLG)”, now U.S. Pat. No. 10,755,053, (iv) U.S. patent application Ser. No. 15/897,364, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Runtime Computation of Story Outlines to Support Natural Language Generation (NLG)”, now U.S. Pat. No. 10,572,606, (v) U.S. patent application Ser. No. 15/897,373, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Ontology Building to Support Natural Language Generation (NLG) Using Composable Communication Goals”, now U.S. Pat. No. 10,719,542, and (vi) U.S. patent application Ser. No. 15/897,381, filed Feb. 15, 2018, entitled “Applied Artificial Intelligence Technology for Interactive Story Editing to Support Natural Language Generation (NLG)”, now U.S. Pat. No. 10,713,442, each of which claims priority to U.S. provisional patent application Ser. No. 62/460,349, filed Feb. 17, 2017, and entitled “Applied Artificial Intelligence Technology for Performing Natural Language Generation (NLG) Using Composable Communication Goals and Ontologies to Generate Narrative Stories”, the entire disclosures of each of which are incorporated herein by reference. This patent application is related to U.S. patent application Ser. No. 16/047,800, filed Jul. 27, 2018, entitled “Applied Artificial Intelligence Technology for Narrative Generation Based on Analysis Communication Goals”, now U.S. Pat. No. 10,699,079, the entire disclosure of which is incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4992939 | Tyler | Feb 1991 | A |
| 5619631 | Schott | Apr 1997 | A |
| 5734916 | Greenfield et al. | Mar 1998 | A |
| 5802495 | Goltra | Sep 1998 | A |
| 6006175 | Holzrichter | Dec 1999 | A |
| 6144938 | Surace et al. | Nov 2000 | A |
| 6278967 | Akers et al. | Aug 2001 | B1 |
| 6289363 | Consolatti et al. | Sep 2001 | B1 |
| 6757362 | Cooper et al. | Jun 2004 | B1 |
| 6771290 | Hoyle | Aug 2004 | B1 |
| 6968316 | Hamilton | Nov 2005 | B1 |
| 6976031 | Toupal et al. | Dec 2005 | B1 |
| 7027974 | Busch et al. | Apr 2006 | B1 |
| 7246315 | Andrieu et al. | Jul 2007 | B1 |
| 7333967 | Bringsjord et al. | Feb 2008 | B1 |
| 7496621 | Pan et al. | Feb 2009 | B2 |
| 7577634 | Ryan et al. | Aug 2009 | B2 |
| 7610279 | Budzik et al. | Oct 2009 | B2 |
| 7617199 | Budzik et al. | Nov 2009 | B2 |
| 7617200 | Budzik et al. | Nov 2009 | B2 |
| 7627565 | Budzik et al. | Dec 2009 | B2 |
| 7644072 | Budzik et al. | Jan 2010 | B2 |
| 7657518 | Budzik et al. | Feb 2010 | B2 |
| 7716116 | Schiller | May 2010 | B2 |
| 7778895 | Baxter et al. | Aug 2010 | B1 |
| 7825929 | Kincaid | Nov 2010 | B2 |
| 7836010 | Hammond et al. | Nov 2010 | B2 |
| 7840448 | Musgrove et al. | Nov 2010 | B2 |
| 7856390 | Schiller | Dec 2010 | B2 |
| 7865496 | Schiller | Jan 2011 | B1 |
| 7930169 | Billerey-Mosier | Apr 2011 | B2 |
| 8046226 | Soble et al. | Oct 2011 | B2 |
| 8311863 | Kemp | Nov 2012 | B1 |
| 8355903 | Birnbaum et al. | Jan 2013 | B1 |
| 8374848 | Birnbaum et al. | Feb 2013 | B1 |
| 8442940 | Faletti et al. | May 2013 | B1 |
| 8447604 | Chang | May 2013 | B1 |
| 8463695 | Schiller | Jun 2013 | B2 |
| 8494944 | Schiller | Jul 2013 | B2 |
| 8515737 | Allen | Aug 2013 | B2 |
| 8612208 | Cooper et al. | Dec 2013 | B2 |
| 8630844 | Nichols et al. | Jan 2014 | B1 |
| 8630912 | Seki et al. | Jan 2014 | B2 |
| 8630919 | Baran et al. | Jan 2014 | B2 |
| 8645825 | Cornea et al. | Feb 2014 | B1 |
| 8676691 | Schiller | Mar 2014 | B2 |
| 8688434 | Birnbaum et al. | Apr 2014 | B1 |
| 8751563 | Warden et al. | Jun 2014 | B1 |
| 8762133 | Reiter | Jun 2014 | B2 |
| 8762134 | Reiter | Jun 2014 | B2 |
| 8775161 | Nichols et al. | Jul 2014 | B1 |
| 8812311 | Weber | Aug 2014 | B2 |
| 8843363 | Birnbaum et al. | Sep 2014 | B2 |
| 8886520 | Nichols et al. | Nov 2014 | B1 |
| 8892417 | Nichols et al. | Nov 2014 | B1 |
| 8892419 | Lundberg et al. | Nov 2014 | B2 |
| 8903711 | Lundberg et al. | Dec 2014 | B2 |
| 9135244 | Reiter | Sep 2015 | B2 |
| 9208147 | Nichols et al. | Dec 2015 | B1 |
| 9244894 | Dale et al. | Jan 2016 | B1 |
| 9251134 | Birnbaum et al. | Feb 2016 | B2 |
| 9323743 | Reiter | Apr 2016 | B2 |
| 9336193 | Logan et al. | May 2016 | B2 |
| 9355093 | Reiter | May 2016 | B2 |
| 9396168 | Birnbaum et al. | Jul 2016 | B2 |
| 9396181 | Sripada et al. | Jul 2016 | B1 |
| 9396758 | Oz et al. | Jul 2016 | B2 |
| 9405448 | Reiter | Aug 2016 | B2 |
| 9424254 | Howald et al. | Aug 2016 | B2 |
| 9430557 | Bhat et al. | Aug 2016 | B2 |
| 9460075 | Mungi et al. | Oct 2016 | B2 |
| 9483520 | Reiner et al. | Nov 2016 | B1 |
| 9529795 | Kondadadi et al. | Dec 2016 | B2 |
| 9569729 | Oehrle et al. | Feb 2017 | B1 |
| 9576009 | Hammond | Feb 2017 | B1 |
| 9665259 | Lee et al. | May 2017 | B2 |
| 9697178 | Nichols et al. | Jul 2017 | B1 |
| 9697197 | Birnbaum et al. | Jul 2017 | B1 |
| 9697492 | Birnbaum et al. | Jul 2017 | B1 |
| 9720884 | Birnbaum et al. | Aug 2017 | B2 |
| 9720899 | Birnbaum et al. | Aug 2017 | B1 |
| 9870629 | Cardno et al. | Jan 2018 | B2 |
| 9946711 | Reiter et al. | Apr 2018 | B2 |
| 9971967 | Bufe, III et al. | May 2018 | B2 |
| 9977773 | Birnbaum et al. | May 2018 | B1 |
| 9990337 | Birnbaum et al. | Jun 2018 | B2 |
| 10019512 | Boyle et al. | Jul 2018 | B2 |
| 10037377 | Boyle et al. | Jul 2018 | B2 |
| 10049152 | Ajmera et al. | Aug 2018 | B2 |
| 10162900 | Chatterjee et al. | Dec 2018 | B1 |
| 10185477 | Paley et al. | Jan 2019 | B1 |
| 10387970 | Wang et al. | Aug 2019 | B1 |
| 10416841 | Riche et al. | Sep 2019 | B2 |
| 10482381 | Nichols et al. | Nov 2019 | B2 |
| 10489488 | Birnbaum et al. | Nov 2019 | B2 |
| 10565308 | Reiter | Feb 2020 | B2 |
| 10572606 | Paley et al. | Feb 2020 | B1 |
| 10585983 | Paley et al. | Mar 2020 | B1 |
| 10621183 | Chatterjee et al. | Apr 2020 | B1 |
| 10657201 | Nichols et al. | May 2020 | B1 |
| 10699079 | Paley | Jun 2020 | B1 |
| 10706236 | Platt et al. | Jul 2020 | B1 |
| 10747823 | Birnbaum et al. | Aug 2020 | B1 |
| 10755042 | Birnbaum et al. | Aug 2020 | B2 |
| 10755046 | Lewis Meza et al. | Aug 2020 | B1 |
| 10762304 | Paley et al. | Sep 2020 | B1 |
| 10853583 | Platt et al. | Dec 2020 | B1 |
| 10943069 | Paley | Mar 2021 | B1 |
| 10956656 | Birnbaum et al. | Mar 2021 | B2 |
| 10963649 | Sippel et al. | Mar 2021 | B1 |
| 10990767 | Smathers et al. | Apr 2021 | B1 |
| 11003866 | Sippel et al. | May 2021 | B1 |
| 11023689 | Sippel et al. | Jun 2021 | B1 |
| 11030408 | Meza et al. | Jun 2021 | B1 |
| 11042708 | Pham et al. | Jun 2021 | B1 |
| 11042709 | Pham et al. | Jun 2021 | B1 |
| 11042713 | Platt et al. | Jun 2021 | B1 |
| 11068661 | Nichols et al. | Jul 2021 | B1 |
| 11126798 | Lewis Meza et al. | Sep 2021 | B1 |
| 11144838 | Platt et al. | Oct 2021 | B1 |
| 11170038 | Platt et al. | Nov 2021 | B1 |
| 11182556 | Lewis Meza et al. | Nov 2021 | B1 |
| 11188588 | Platt et al. | Nov 2021 | B1 |
| 11222184 | Platt et al. | Jan 2022 | B1 |
| 11232268 | Platt et al. | Jan 2022 | B1 |
| 11232270 | Platt et al. | Jan 2022 | B1 |
| 11238090 | Platt et al. | Feb 2022 | B1 |
| 11288328 | Birnbaum et al. | Mar 2022 | B2 |
| 11334726 | Platt et al. | May 2022 | B1 |
| 11341330 | Smathers et al. | May 2022 | B1 |
| 11341338 | Platt et al. | May 2022 | B1 |
| 20020046018 | Marcu et al. | Apr 2002 | A1 |
| 20020083025 | Robarts et al. | Jun 2002 | A1 |
| 20020107721 | Darwent et al. | Aug 2002 | A1 |
| 20030004706 | Yale et al. | Jan 2003 | A1 |
| 20030110186 | Markowski et al. | Jun 2003 | A1 |
| 20040015342 | Garst | Jan 2004 | A1 |
| 20040138899 | Birnbaum et al. | Jul 2004 | A1 |
| 20040174397 | Cereghini et al. | Sep 2004 | A1 |
| 20040225651 | Musgrove et al. | Nov 2004 | A1 |
| 20040230989 | Macey et al. | Nov 2004 | A1 |
| 20040255232 | Hammond et al. | Dec 2004 | A1 |
| 20050027704 | Hammond et al. | Feb 2005 | A1 |
| 20050028156 | Hammond et al. | Feb 2005 | A1 |
| 20050033582 | Gadd et al. | Feb 2005 | A1 |
| 20050125213 | Chen et al. | Jun 2005 | A1 |
| 20050273362 | Harris et al. | Dec 2005 | A1 |
| 20060031182 | Ryan et al. | Feb 2006 | A1 |
| 20060101335 | Pisciottano | May 2006 | A1 |
| 20060212446 | Hammond et al. | Sep 2006 | A1 |
| 20060218485 | Blumenthal | Sep 2006 | A1 |
| 20060253431 | Bobick et al. | Nov 2006 | A1 |
| 20060253783 | Vronay et al. | Nov 2006 | A1 |
| 20060271535 | Hammond et al. | Nov 2006 | A1 |
| 20060277168 | Hammond et al. | Dec 2006 | A1 |
| 20070132767 | Wright et al. | Jun 2007 | A1 |
| 20070136657 | Blumenthal et al. | Jun 2007 | A1 |
| 20070185846 | Budzik et al. | Aug 2007 | A1 |
| 20070185847 | Budzik et al. | Aug 2007 | A1 |
| 20070185861 | Budzik et al. | Aug 2007 | A1 |
| 20070185862 | Budzik et al. | Aug 2007 | A1 |
| 20070185863 | Budzik et al. | Aug 2007 | A1 |
| 20070185864 | Budzik et al. | Aug 2007 | A1 |
| 20070185865 | Budzik et al. | Aug 2007 | A1 |
| 20070250479 | Lunt et al. | Oct 2007 | A1 |
| 20070250826 | O'Brien | Oct 2007 | A1 |
| 20080005677 | Thompson | Jan 2008 | A1 |
| 20080140696 | Mathuria | Jun 2008 | A1 |
| 20080198156 | Jou et al. | Aug 2008 | A1 |
| 20080250070 | Abdulla et al. | Oct 2008 | A1 |
| 20080256066 | Zuckerman et al. | Oct 2008 | A1 |
| 20080304808 | Newell et al. | Dec 2008 | A1 |
| 20080306882 | Schiller | Dec 2008 | A1 |
| 20080313130 | Hammond et al. | Dec 2008 | A1 |
| 20090019013 | Tareen et al. | Jan 2009 | A1 |
| 20090030899 | Tareen et al. | Jan 2009 | A1 |
| 20090049041 | Tareen et al. | Feb 2009 | A1 |
| 20090083288 | LeDain et al. | Mar 2009 | A1 |
| 20090089100 | Nenov et al. | Apr 2009 | A1 |
| 20090144608 | Oisel et al. | Jun 2009 | A1 |
| 20100146393 | Land et al. | Jun 2010 | A1 |
| 20100161541 | Covannon et al. | Jun 2010 | A1 |
| 20100185984 | Wright et al. | Jul 2010 | A1 |
| 20100325107 | Kenton et al. | Dec 2010 | A1 |
| 20110044447 | Morris et al. | Feb 2011 | A1 |
| 20110078105 | Wallace | Mar 2011 | A1 |
| 20110087486 | Schiller | Apr 2011 | A1 |
| 20110113315 | Datha et al. | May 2011 | A1 |
| 20110113334 | Joy et al. | May 2011 | A1 |
| 20110213642 | Makar et al. | Sep 2011 | A1 |
| 20110246182 | Allen | Oct 2011 | A1 |
| 20110249953 | Suri et al. | Oct 2011 | A1 |
| 20110261049 | Cardno et al. | Oct 2011 | A1 |
| 20110295903 | Chen | Dec 2011 | A1 |
| 20110307435 | Overell et al. | Dec 2011 | A1 |
| 20110311144 | Tardif | Dec 2011 | A1 |
| 20110314381 | Fuller et al. | Dec 2011 | A1 |
| 20120069131 | Abelow | Mar 2012 | A1 |
| 20120109637 | Merugu et al. | May 2012 | A1 |
| 20120143849 | Wong et al. | Jun 2012 | A1 |
| 20120158850 | Harrison et al. | Jun 2012 | A1 |
| 20120166180 | Au | Jun 2012 | A1 |
| 20120265531 | Bennett | Oct 2012 | A1 |
| 20120310699 | McKenna et al. | Dec 2012 | A1 |
| 20130041677 | Nusimow et al. | Feb 2013 | A1 |
| 20130091031 | Baran et al. | Apr 2013 | A1 |
| 20130096947 | Shah et al. | Apr 2013 | A1 |
| 20130144605 | Brager et al. | Jun 2013 | A1 |
| 20130144606 | Birnbaum et al. | Jun 2013 | A1 |
| 20130145242 | Birnbaum et al. | Jun 2013 | A1 |
| 20130173285 | Hyde et al. | Jul 2013 | A1 |
| 20130174026 | Locke | Jul 2013 | A1 |
| 20130187926 | Silverstein et al. | Jul 2013 | A1 |
| 20130211855 | Eberle et al. | Aug 2013 | A1 |
| 20130238330 | Casella dos Santos | Sep 2013 | A1 |
| 20130246300 | Fischer et al. | Sep 2013 | A1 |
| 20130253910 | Turner et al. | Sep 2013 | A1 |
| 20130262092 | Wasick | Oct 2013 | A1 |
| 20130275121 | Tunstall-Pedoe | Oct 2013 | A1 |
| 20130304507 | Dail et al. | Nov 2013 | A1 |
| 20130316834 | Vogel et al. | Nov 2013 | A1 |
| 20140006012 | Zhou et al. | Jan 2014 | A1 |
| 20140040312 | Gorman et al. | Feb 2014 | A1 |
| 20140062712 | Reiter | Mar 2014 | A1 |
| 20140075004 | Van Dusen et al. | Mar 2014 | A1 |
| 20140134590 | Hiscock, Jr. | May 2014 | A1 |
| 20140173425 | Hailpern et al. | Jun 2014 | A1 |
| 20140201202 | Jones et al. | Jul 2014 | A1 |
| 20140208215 | Deshpande | Jul 2014 | A1 |
| 20140314225 | Riahi et al. | Oct 2014 | A1 |
| 20140351281 | Tunstall-Pedoe | Nov 2014 | A1 |
| 20140356833 | Sabczynski et al. | Dec 2014 | A1 |
| 20140372850 | Campbell et al. | Dec 2014 | A1 |
| 20140375466 | Reiter | Dec 2014 | A1 |
| 20150049951 | Chaturvedi et al. | Feb 2015 | A1 |
| 20150078232 | Djinki et al. | Mar 2015 | A1 |
| 20150088808 | Tyagi et al. | Mar 2015 | A1 |
| 20150134694 | Burke et al. | May 2015 | A1 |
| 20150142704 | London | May 2015 | A1 |
| 20150161997 | Wetsel et al. | Jun 2015 | A1 |
| 20150169548 | Reiter | Jun 2015 | A1 |
| 20150178386 | Oberkampf et al. | Jun 2015 | A1 |
| 20150186504 | Gorman et al. | Jul 2015 | A1 |
| 20150227508 | Howald et al. | Aug 2015 | A1 |
| 20150227588 | Shapira et al. | Aug 2015 | A1 |
| 20150242384 | Reiter | Aug 2015 | A1 |
| 20150261745 | Song et al. | Sep 2015 | A1 |
| 20150268930 | Lee et al. | Sep 2015 | A1 |
| 20150324347 | Bradshaw et al. | Nov 2015 | A1 |
| 20150324351 | Sripada et al. | Nov 2015 | A1 |
| 20150324374 | Sripada et al. | Nov 2015 | A1 |
| 20150325000 | Sripada | Nov 2015 | A1 |
| 20150331846 | Guggilla et al. | Nov 2015 | A1 |
| 20150331850 | Ramish | Nov 2015 | A1 |
| 20150332665 | Mishra et al. | Nov 2015 | A1 |
| 20150347400 | Sripada | Dec 2015 | A1 |
| 20150347901 | Cama et al. | Dec 2015 | A1 |
| 20150356967 | Byron et al. | Dec 2015 | A1 |
| 20150363364 | Sripada | Dec 2015 | A1 |
| 20160019200 | Allen | Jan 2016 | A1 |
| 20160026253 | Bradski et al. | Jan 2016 | A1 |
| 20160027125 | Bryce | Jan 2016 | A1 |
| 20160054889 | Hadley et al. | Feb 2016 | A1 |
| 20160103559 | Maheshwari et al. | Apr 2016 | A1 |
| 20160132489 | Reiter | May 2016 | A1 |
| 20160140090 | Dale et al. | May 2016 | A1 |
| 20160162582 | Chatterjee et al. | Jun 2016 | A1 |
| 20160196491 | Chandrasekaran et al. | Jul 2016 | A1 |
| 20160217133 | Reiter et al. | Jul 2016 | A1 |
| 20160232152 | Mahamood | Aug 2016 | A1 |
| 20160232221 | McCloskey et al. | Aug 2016 | A1 |
| 20160314121 | Arroyo et al. | Oct 2016 | A1 |
| 20170017897 | Bugay et al. | Jan 2017 | A1 |
| 20170024465 | Yeh et al. | Jan 2017 | A1 |
| 20170026705 | Yeh et al. | Jan 2017 | A1 |
| 20170046016 | Riche et al. | Feb 2017 | A1 |
| 20170060857 | Imbruce et al. | Mar 2017 | A1 |
| 20170061093 | Amarasingham et al. | Mar 2017 | A1 |
| 20170068551 | Vadodaria | Mar 2017 | A1 |
| 20170091291 | Bostick et al. | Mar 2017 | A1 |
| 20170116327 | Gorelick et al. | Apr 2017 | A1 |
| 20170140405 | Gottemukkala et al. | May 2017 | A1 |
| 20170185674 | Tonkin et al. | Jun 2017 | A1 |
| 20170213157 | Bugay et al. | Jul 2017 | A1 |
| 20170228372 | Moreno et al. | Aug 2017 | A1 |
| 20170270105 | Ninan et al. | Sep 2017 | A1 |
| 20170293864 | Oh et al. | Oct 2017 | A1 |
| 20170358295 | Roux et al. | Dec 2017 | A1 |
| 20170371856 | Can et al. | Dec 2017 | A1 |
| 20180025726 | Gatti de Bayser et al. | Jan 2018 | A1 |
| 20180089177 | Cho | Mar 2018 | A1 |
| 20180232443 | Delgo et al. | Aug 2018 | A1 |
| 20180234442 | Luo et al. | Aug 2018 | A1 |
| 20180260380 | Birnbaum et al. | Sep 2018 | A1 |
| 20180261203 | Zoller et al. | Sep 2018 | A1 |
| 20180285324 | Birnbaum et al. | Oct 2018 | A1 |
| 20180293483 | Abramson et al. | Oct 2018 | A1 |
| 20180300311 | Krishnamurthy | Oct 2018 | A1 |
| 20180314689 | Wang et al. | Nov 2018 | A1 |
| 20190197097 | Nagarajan et al. | Jun 2019 | A1 |
| 20190236140 | Canim et al. | Aug 2019 | A1 |
| 20190332666 | Dadachev et al. | Oct 2019 | A1 |
| 20200074310 | Li et al. | Mar 2020 | A1 |
| 20200089735 | Birnbaum et al. | Mar 2020 | A1 |
| 20210192132 | Birnbaum et al. | Jun 2021 | A1 |
| 20210192144 | Paley et al. | Jun 2021 | A1 |
| 20220114206 | Platt et al. | Apr 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 9630844 | Oct 1996 | WO |
| 2006122329 | Nov 2006 | WO |
| 2014035400 | Mar 2014 | WO |
| 2014035402 | Mar 2014 | WO |
| 2014035403 | Mar 2014 | WO |
| 2014035406 | Mar 2014 | WO |
| 2014035407 | Mar 2014 | WO |
| 2014035447 | Mar 2014 | WO |
| 2014070197 | May 2014 | WO |
| 2014076524 | May 2014 | WO |
| 2014076525 | May 2014 | WO |
| 2014102568 | Jul 2014 | WO |
| 2014102569 | Jul 2014 | WO |
| 2014111753 | Jul 2014 | WO |
| 2015028844 | Mar 2015 | WO |
| 2015159133 | Oct 2015 | WO |
| Entry |
|---|
| D. L. Roberts and C. L. Isbell, “Lessons on Using Computationally Generated Influence for Shaping Narrative Experiences,” in IEEE Transactions on Computational Intelligence and AI in Games, vol. 6, No. 2, pp. 188-202, Jun. 2014, doi: 10.1109/TCIAIG.2013.2287154. (Year: 2014). |
| G. A. Mack, K. Lonergan, J. Scholtz, M. P. Steves and C. R. Hale, “A framework for metrics in large complex systems,” 2004 IEEE Aerospace Conference Proceedings (IEEE Cat. No. 04TH8720), Big Sky, MT, 2004, pp. 3217-3228 vol. 5, doi: 10.1109/AERO.2004.1368127. (Year: 2004) (Year: 2004). |
| W. W. Smari, K. Weigand, G. Petonito, Y. Kantamani, R. Madala and S. Donepudi, “An integrated approach to collaborative decision making using computer-supported conflict management methodology,” IRI—2005 IEEE International Conference on Information Reuse and Integration, Conf, 2005., 2005, pp. 182-219 (Year: 2005). |
| D. L. Roberts and C. L. Isbell, “Lessons on Using Computationally Generated Influence for Shaping Narrative Experiences,” in IEEE Transactions on Computational Intelligence and AI in Games, vol. 6, No. 2, pp. 188-202, Jun. 2014, doi: 10.1109/TCIAIG.2013.2287154. (Year: 2014) (Year: 2014). |
| G. A. Mack, K. Lonergan, J. Scholtz, M. P. Steves and C. R. Hale, “A framework for metrics in large complex systems,” 2004 IEEE Aerospace Conference Proceedings (IEEE Cat. No. 04TH8720), Big Sky, MT, 2004, pp. 3217-3228 vol. 5, doi: 10.1109/AERO.2004.1368127. (Year: 2004) (Year: 2004) (Year: 2004). |
| Allen et al., “StatsMonkey: A Data-Driven Sports Narrative Writer”, Computational Models of Narrative: Papers from the AAAI Fall Symposium, Nov. 2010, 2 pages. |
| Andersen, P., Hayes, P., Huettner, A., Schmandt, L., Nirenburg, I., and Weinstein, S. (1992). Automatic extraction of facts from press releases to generate news stories. In Proceedings of the third conference on Applied natural language processing. (Trento, Italy). ACM Press, New York, NY, 170-177. |
| Andre, E., Herzog, G., & Rist, T. (1988). On the simultaneous interpretation of real world image sequences and their natural language description: the system SOCCER. Paper presented at Proceedings of the 8th. European Conference on Artificial Intelligence (ECAI), Munich. |
| Asset Economics, Inc. (Feb. 11, 2011). |
| Bailey, P. (1999). Searching for Storiness: Story-Generation from a Reader's Perspective. AAAI Technical Report FS-99-01. |
| Bethem, T., Burton, J., Caldwell, T., Evans, M., Kittredge, R., Lavoie, B., and Werner, J. (2005). Generation of Realtime Narrative Summaries for Real-time Water Levels and Meteorological Observations in PORTS®. In Proceedings of the Fourth Conference on Artificial Intelligence Applications to Environmental Sciences (AMS-2005), San Diego, California. |
| Bourbeau, L., Carcagno, D., Goldberg, E., Kittredge, R., & Polguere, A. (1990). Bilingual generation of weather forecasts in an operations environment. Paper presented at Proceedings of the 13th International Conference on Computational Linguistics (COLING), Helsinki, Finland, pp. 318-320. |
| Boyd, S. (1998). TREND: a system for generating intelligent descriptions of time series data. Paper presented at Proceedings of the IEEE international conference on intelligent processing systems (ICIPS-1998). |
| Character Writer Version 3.1, Typing Chimp Software LLC, 2012, screenshots from working program, pp. 1-19. |
| Cyganiak et al., “RDF 1.1 Concepts and Abstract Syntax”, W3C Recommendation, 2014, vol. 25, No. 2. |
| Dehn, N. (1981). Story generation after TALE-SPIN. In Proceedings of the Seventh International Joint Conference on Artificial Intelligence. (Vancouver, Canada). |
| Dramatica Pro version 4, Write Brothers, 1993-2006, user manual. |
| Gatt, A., and Portet, F. (2009). Text content and task performance in the evaluation of a Natural Language Generation System. Proceedings of the Conference on Recent Advances in Natural Language Processing (RANLP-09). |
| Gatt, A., Portet, F., Reiter, E., Hunter, J., Mahamood, S., Moncur, W., and Sripada, S. (2009). From data to text in the Neonatal Intensive Care Unit: Using NLG technology for decision support and information management. AI Communications 22, pp. 153-186. |
| Glahn, H. (1970). Computer-produced worded forecasts. Bulletin of the American Meteorological Society, 51(12), 1126-1131. |
| Goldberg, E., Driedger, N., & Kittredge, R. (1994). Using Natural-Language Processing to Produce Weather Forecasts. IEEE Expert, 9 (2), 45. |
| Hargood, C., Millard, D. and Weal, M. (2009) Exploring the Importance of Themes in Narrative Systems. |
| Hargood, C., Millard, D. and Weal, M. (2009). Investigating a Thematic Approach to Narrative Generation, 2009. |
| Hunter, J., Freer, Y., Gatt, A., Logie, R., McIntosh, N., van der Meulen, M., Portet, F., Reiter, E., Sripada, S., and Sykes, C. (2008). Summarising Complex ICU Data in Natural Language. AMIA 2008 Annual Symposium Proceedings, pp. 323-327. |
| Hunter, J., Gatt, A., Portet, F., Reiter, E., and Sripada, S. (2008). Using natural language generation technology to improve information flows in intensive care units. Proceedings of the 5th Conference on Prestigious Applications of Intelligent Systems, PAIS-08. |
| Kittredge, R., and Lavoie, B. (1998). MeteoCogent: A Knowledge-Based Tool for Generating Weather Forecast Texts. In Proceedings of the American Meteorological Society AI Conference (AMS-98), Phoenix, Arizona. |
| Kittredge, R., Polguere, A., & Goldberg, E. (1986). Synthesizing weather reports from formatted data. Paper presented at Proceedings of the 11th International Conference on Computational Linguistics, Bonn, Germany, pp. 563-565. |
| Kukich, K. (1983). Design of a Knowledge-Based Report Generator. Proceedings of the 21st Conference of the Association for Computational Linguistics, Cambridge, MA, pp. 145-150. |
| Kukich, K. (1983). Knowledge-Based Report Generation: A Technique for Automatically Generating Natural Language Reports from Databases. Paper presented at Proceedings of the Sixth International ACM SIGIR Conference, Washington, DC. |
| Mack et al., “A Framework for Metrics in Large Complex Systems”, IEEE Aerospace Conference Proceedings, 2004, pp. 3217-3228, vol. 5, doi: 10 1109/AERO .2004.1368127. |
| McKeown, K., Kukich, K., & Shaw, J. (1994). Practical issues in automatic documentation generation. 4th Conference on Applied Natural Language Processing, Stuttgart, Germany, pp. 7-14. |
| Meehan, James R., Tale-Spin. (1977). An Interactive Program that Writes Stories. In Proceedings of the Fifth International Joint Conference on Artificial Intelligence. |
| Memorandum Opinion and Order for O2 Media, LLC v. Narrative Science Inc., Case 1:15-cv-05129 (N.D. IL), Feb. 25, 2016, 25 pages (invalidating claims of U.S. Pat. Nos. 7,856,390, 8,494,944, and 8,676,691 owned by O2 Media, LLC. |
| Moncur, W., and Reiter, E. (2007). How Much to Tell? Disseminating Affective Information across a Social Network. Proceedings of Second International Workshop on Personalisation for e-Health. |
| Moncur, W., Masthoff, J., Reiter, E. (2008) What Do You Want to Know? Investigating the Information Requirements of Patient Supporters. 21st IEEE International Symposium on Computer-Based Medical Systems (CBMS 2008), pp. 443-448. |
| Movie Magic Screenwriter, Write Brothers, 2009, user manual. |
| Notice of Allowance for U.S. Appl. No. 16/047,800 dated Feb. 18, 2020. |
| Office Action for U.S. Appl. No. 15/897,331 dated Mar. 25, 2019. |
| Office Action for U.S. Appl. No. 15/897,350 dated Mar. 25, 2019. |
| Portet, F., Reiter, E., Gatt, A., Hunter, J., Sripada, S., Freer, Y., and Sykes, C. (2009). Automatic Generation of Textual Summaries from Neonatal Intensive Care Data. Artificial Intelligence. |
| Portet, F., Reiter, E., Hunter, J., and Sripada, S. (2007). Automatic Generation of Textual Summaries from Neonatal Intensive Care Data. In: Bellazzi, Riccardo, Ameen Abu-Hanna and Jim Hunter (Ed.), 11th Conference on Artificial Intelligence in Medicine (AIME 07), pp. 227-236. |
| Prosecution History for U.S. Appl. No. 16/047,800, now U.S. Pat. No. 10,699,079, filed Jul. 27, 2018. |
| Reiter et al., “Building Applied Natural Generation Systems”, Cambridge University Press, 1995, pp. 1-32. |
| Reiter, E. (2007). An architecture for Data-to-Text systems. In: Busemann, Stephan (Ed.), Proceedings of the 11th European Workshop on Natural Language Generation, pp. 97-104. |
| Reiter, E., Gatt, A., Portet, F., and van der Meulen, M. (2008). The importance of narrative and other lessons from an evaluation of an NLG system that summarises clinical data. Proceedings of the 5th International Conference on Natural Language Generation. |
| Reiter, E., Sripada, S., Hunter, J., Yu, J., and Davy, I. (2005). Choosing words in computer-generated weather forecasts. Artificial Intelligence, 167:137-169. |
| Riedl et al., “From Linear Story Generation to Branching Story Graphs”, IEEE Computer Graphics and Applications, 2006, pp. 23-31. |
| Riedl et al., “Narrative Planning: Balancing Plot and Character”, Journal of Artificial Intelligence Research, 2010, pp. 217-268, vol. 39. |
| Roberts et al., “Lessons on Using Computationally Generated Influence for Shaping Narrative Experiences”, IEEE Transactions on Computational Intelligence and AI in Games, Jun. 2014, pp. 188-202, vol. 6, No. 2, doi: 10.1109/TCIAIG .2013.2287154. |
| Robin, J. (1996). Evaluating the portability of revision rules for incremental summary generation. Paper presented at Proceedings of the 34th. Annual Meeting of the Association for Computational Linguistics (ACL'96), Santa Cruz, CA. |
| Rui, Y., Gupta, A., and Acero, A. 2000. Automatically extracting highlights for TV Baseball programs. In Proceedings of the eighth ACM international conference on Multimedia. (Marina del Rey, California, United States). ACM Press, New York, NY 105-115. |
| Smith, “The Multivariable Method in Singular Perturbation Analysis”, SIAM Review, 1975, pp. 221-273, vol. 17, No. 2. |
| Sripada, S., Reiter, E., and Davy, I. (2003). SumTime-Mousam: Configurable Marine Weather Forecast Generator. Expert Update 6(3):4-10. |
| Storyview, Screenplay Systems, 2000, user manual. |
| Theune, M., Klabbers, E., Odijk, J., dePijper, J., and Krahmer, E. (2001) “From Data to Speech: A General Approach”, Natural Language Engineering 7(1): 47-86. |
| Thomas, K., and Sripada, S. (2007). Atlas.txt: Linking Geo-referenced Data to Text for NLG. Paper presented at Proceedings of the 2007 European Natural Language Generation Workshop (ENLGO7). |
| Thomas, K., and Sripada, S. (2008). What's in a message? Interpreting Geo-referenced Data for the Visually-impaired. Proceedings of the Int. conference on NLG. |
| Thomas, K., Sumegi, L., Ferres, L., and Sripada, S. (2008). Enabling Access to Geo-referenced Information: Atlas.txt. Proceedings of the Cross-disciplinary Conference on Web Accessibility. |
| Van der Meulen, M., Logie, R., Freer, Y., Sykes, C., McIntosh, N., and Hunter, J. (2008). When a Graph is Poorer than 100 Words: A Comparison of Computerised Natural Language Generation, Human Generated Descriptions and Graphical Displays in Neonatal Intensive Care. Applied Cognitive Psychology. |
| Yu, J., Reiter, E., Hunter, J., and Mellish, C. (2007). Choosing the content of textual summaries of large time-series data sets. Natural Language Engineering, 13:25-49. |
| Yu, J., Reiter, E., Hunter, J., and Sripada, S. (2003). Sumtime-Turbine: A Knowledge-Based System to Communicate Time Series Data in the Gas Turbine Domain. In P Chung et al. (Eds) Developments in Applied Artificial Intelligence: Proceedings of IEA/AIE-2003, pp. 379-384. Springer (LNAI 2718). |
| Miahamood et al., “Generating Annotated Graphs Using the NLG Pipeline Architecture”, Proceedings of the 8th International Natural Language Generation Conference (INLG), 2014. |
| Segel et al., “Narrative Visualization: Telling Stories with Data”, Stanford University, Oct. 2010, 10 pgs. |
| Number | Date | Country | |
|---|---|---|---|
| 20210192144 A1 | Jun 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62539832 | Aug 2017 | US | |
| 62460349 | Feb 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16047837 | Jul 2018 | US |
| Child | 17191362 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15897359 | Feb 2018 | US |
| Child | 16047837 | US | |
| Parent | 15897331 | Feb 2018 | US |
| Child | 15897359 | US | |
| Parent | 15897373 | Feb 2018 | US |
| Child | 15897331 | US | |
| Parent | 15897350 | Feb 2018 | US |
| Child | 15897373 | US | |
| Parent | 15897381 | Feb 2018 | US |
| Child | 15897350 | US | |
| Parent | 15897364 | Feb 2018 | US |
| Child | 15897381 | US |
