Patent Application
20250013635
This invention relates to repairing results output by a large language model (LLM).
Large language models (LLM) may be trained to process natural language statements (“prompts”) and generate natural language prose response to the prompts. A LLM may be trained with a large corpus of text and provide sophisticated responses. However, the LLM may often provide results that have the appearance of a correct result but are factually or otherwise in correct. Although further training may eliminate some errors, incorrect results may still appear.
In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the invention will be described and explained with additional specificity and detail through use of the accompanying drawings, in which:
Referring to
Each database 104 may have a schema 106 that defines the logical relationship between tables 108 of the database 104. Each table 108 may include such information as a table identifier 110, a set of keys 112 and values 114 for each key 112. Each table 108 may include multiple columns, such as two or more columns in the form of a column of keys 112 and one or more columns of values for the keys 112.
Each database 104 may be implemented as a structured query language (SQL) database and the server system 102 may be implemented as an SQL server. Other types of databases may be used in a like manner, such as MYSQL, ORACLE database, IBM DB2, AMAZON RELATIONAL DATABASE SERVICE (RDS), POSTGRE SQL, or the like. SQL is referred to the throughout the following description with the understanding that any other type of database and database language may be used in a like manner.
Users at a user computing device 116 may submit queries 118 to the server system 102 by means of a network 120, such as a local area network (LAN), wide area network (WAN), the Internet, or other type of network. Queries may also be submitted by software, such as front-end software for executing ecommerce transactions, interfacing with a client application, or performing other functions. The server system 102 processes the queries with respect to one or more databases 104 referenced by the query and returns a response to the user computing device 116.
Referring to
The query tree 200 may one or more operations 210 to be performed with respect to the one or more sources 204. The operations 210 may include mathematical (addition, subtraction, multiplication, division, etc.), Boolean, or other operations to be performed with respect to the one or more sources 204. One or more inputs to each operation 210 may be one of the one or more sources 204 or the result of another operation 210. The query tree 200 may specify results 212 that identify columns of the results of the operations 210 that are to be provided as the output of the query 118. The results 212 may include names 214 that specify one or both of columns or values that are to be selected as the result and a location at which results of the operations 210 are to be stored.
The method 300 includes collecting 302 queries with respect to the databases 104, such as databases 104 belonging to a common data warehouse of one or more objects. The method 300 may be performed continuously or periodically such that step 302 includes collecting queries 118 submitted to the server system 102 since a last iteration of the method 300 or within a time window (e.g., one month, one year, etc.) preceding execution of the method 300.
The method 300 may include scoring 304 queries according to frequency of occurrence. For example, queries collected at step 302 may be grouped together as being identical to one another. The number of queries in a group may then be used as the score for the queries of the group. Other factors may also contribute to the score, such as a source of the queries. Queries from software such as Tableau may be vetted and of known importance and may be given greater weight, e.g., a single query may be counted as more than one query when counting the number of occurrences of a query.
The method 300 may include creating 306 initial nodes for a knowledge graph from the queries collected at step 302. For example, for each query (which may represent a group of queries), the nodes created for the query may include nodes representing some or all of:
Each node may be represented by a data structure describing the object (table, operation, result, query, etc.) represented by the node. For example, a node representing a table may include the table identifier 110 of the table. A node representing the operation (Ship Date-Order Date) may include data referencing a Ship Date table, an Order Date table, and the mathematical subtraction operation. The Ship Date table and Order Date table themselves may represent the result of a database operation, such as a JOIN operation that itself may be represented as a node.
The method 300 may include creating node connections from the queries selected at step 302. For example, connections between nodes may have some or all of the following types:
Each connection may be represented by a data structure that includes identifiers of the nodes connected by the connection and data describing the type of the connection. Note that a table may be referenced by multiple queries. Accordingly, connections created at step 308 may likewise connect nodes created from multiple queries.
Each node and connection may have a score assigned 310 thereto based on the scores from step 304. For example, for a table, the score may correspond to the total number of queries referencing the table. For nodes and connections created from queries grouped together as being identical, the nodes and connections may have scores assigned thereto equal to the number of queries in the group.
Referring to
The method 400 may include consolidating 402 one or both of the nodes and connections of the initial version of the knowledge graph. For example, step 402 may include identifying queries that are sufficiently identical in terms of structure of operations 206, 210. Step 402 may include identifying parts of queries that are identical in terms of structures of operations 206, 210. For example, patterns of nodes and connections that are similar may be identified. As noted above, each node includes information indicating the table, operation 206, 210, or query represented by the node and each connection includes information describing the type of the connection. Accordingly, groups of identical nodes with identical connections may be identified and consolidated.
For example, referring to
As an example of the approach described above with respect to
The method 400 may include performing 404 statistical analysis of the underlying data and determining 406 relationship information based on the statistical analysis. For example, for a node representing a table, statistical information concerning the values in the table may be determined. For example, the statistical information may include whether values in a column of the table are unique relative to one another and possibly unique relative to values in other columns of the table or other tables.
In a first example, each customer identifier in a customer identifier table may be unique whereas different customer identifiers may have the same birth date from a birth date table, thereby indicating that the birth date table is a dimension of the customer identifier table.
In a second example, if there is a JOIN operation between a key of a first table and a column of a second table, the node representing the first table may be deemed to be a key node whereas the second table is deemed a dimension node of the key node.
In a third example, in a table including order identifiers, all of the order identifiers may be unique whereas other values such as dates, states, zip codes, and product identifiers, price, etc. may be the same in multiple rows. Accordingly, the order identifier may be chosen to be a key node whereas tables of dates, states, zip codes, product identifiers, price, etc. may be deemed dimension nodes having a dimension relationship to the key node.
In a fourth example, a node may represent a join of an order identifier table, a sold date table, and item identifier table. Since the sold date for each item in the same order is the same, the date, as represented by the date table, may be deemed a dimension of the order identifier, as represented by the order identifier table.
In a fifth example, or a given combination of an order identifier and item identifier (i.e., the same order), all prices are constant. Accordingly, for a node representing the combination (e.g., a joint of columns or tables for the order identifier and item identifier), the price, as represented by a price table, may be a dimension. The combination may be identified due to one or more queries performing a JOIN on two tables or columns of tables.
Many databases have a schema 106, such as databases generated by or provided by a software as a service (SaaS) company, such as SALESFORCE, MARQETA, SEGMENT, or the like. The schema 106 may indicate the meaning of data stored in tables and the relationships between tables, such as the meaning of the data stored in each column of each table. Accordingly, the method 400 may include determining 408 relationships from any schemas 106 for the databases 104 hosting tables 108 referenced by the initial knowledge graph, including which nodes are key nodes according to the schemas 106 and which nodes are dimension nodes to a given key node.
The initial knowledge graph may then be augmented 410 with the relationship information from one or both of steps 406 and 408. For example, a connection between a key node and a dimension node as determined at step 406 and 408 may be augmented with information indicating that the connection represents a dimension relationship and the direction of the relationship, i.e., which node is the dimension node and which node is the key node. Other type of relationships, such as between key nodes, may be a “one-to-many” relationship, such as between a key node representing a customer identifier and a key node representing customer orders since a single customer may have multiple orders.
The entity graph may be derived from the final knowledge graph by pruning 502 connections with low statistical significance. As noted above, connections may be assigned a score based on the number of queries represented by the connection. Accordingly, connections with a low score are not of high interest to users of the databases 104 and may be pruned. Connections with low statistical significance may be those with a score below a predetermined threshold. Connections with low statistical significance may be X percent of the connections with the lowest scores, where X is a predetermined value, such as a value between 0 and 90, 0 and 75, 0 and 50, or 0 and 25.
The method 500 may include selecting 504 nodes of the knowledge graph as entities of the entity graph according to evaluation of the connections between nodes of the knowledge graph and adding 506 nodes from the knowledge graph to the entities as dimensions of the entities according to connections of the knowledge graph. For example, a node representing a customer identifier may be an entity whereas nodes representing attributes of a customer (address, state, orders, etc.) are added as dimensions. In another example, a node representing an order identifier node may be selected as an entity whereas nodes representing tables for purchase date, ship date, item identifiers, purchase price, or other attributes of an order are added as dimensions of the entity.
In one example, key nodes are selected as entities for the entity graph and dimension nodes are added to the entities as dimensions. In another example, only a subset of key nodes are selected as entities, such as the top Y with the highest score, where X is a predetermined value, such as a value between 10 and 20, 20 and 50, 50 and 80, or 80 and 100.
For each entity created for a key node, dimensions may be added to the entity for each dimension node of the knowledge graph connected to the entity. Connections between entities may be retained in the entity graph, i.e., connections between key nodes of the knowledge graph.
To further enhance usability of the entity graph, entities and possibly dimensions may be processed 508 by a logical learning model (LLM) to assign human-intelligible labels to the entities of the entity graph. The LLM may be an LLM trained to perform the labeling of step 508 or may be a general purpose LLM such as CHATGPT, BARD, or the like.
For example, for an entity, the LLM may receive the name of the object in the databases 104 (e.g., table identifier 110). The LLM may additionally or alternatively receive underlying data for the entity and possibly the objects represented by dimensions of the entity, such as the contents of the table represented by the entity or dimension of the entity, the results of an operation represented by the entity or dimension of the entity, or other information.
The LLM outputs a human-intelligible label based on the inputs that captures the concept represented by the entity. For example, a database table 108 for customer identifiers may be labeled CK_customer_db and have dimensions such as 2022_order_db, NA_address_db, or any arbitrary names. The names of tables represented by entities and dimensions may even be arbitrary codes, e.g., GX_2023_CM. By evaluating the identifiers, the underlying data (e.g., orders, addresses, birth date, and other human-related information), the LLM may derive that a table can be represented by the label “customer.”
In some embodiments, the LLM may provide superior results if the entity graph in its entirety, or at least groups of multiple entities and their corresponding dimensions, is submitted to the LLM with the task of assigning unique names to each entity (or possibly to each entity and each dimension).
Referring to
Referring to
The knowledge graph generated according to the approach described with respect to
For example, a user may wish for information about “customers who were born in Texas.” This natural language statement can be readily expressed by the user in KGL as “customers where CurrentAddressState=‘TX.’” As another example, a user may wish for information about “Customers who have more than one order.” This natural language statement can be expressed in KGL as “customers where count (orders)>1.”
The KGL may have at least some of the following properties:
The KGL may be constrained with respect to a knowledge graph, such that variables included in a KGL statement are constrained to be entities 602 or dimensions 604 of the entity graph. However, inasmuch as the entities 602 and dimensions 604 are assigned human-intelligible labels, usage of the knowledge graph is made simple. For example, a KGL editor may provide a graphical representation of the entity graph in the form of a directory structure or other graphical representation of the hierarchical structure of the entity graph. A user may therefore select view entities in the entity graph and select therefrom. Likewise, interface aids such as autocomplete may provide suggestions for entities in the entity graph in response to a string input by a user. A search interface to the entity graph may further enable a user to search for potentially relevant entities and dimensions from the entity graph based on a natural language search string. For example, a search engine may enable a user search for an entity matching a natural language search string based on lexical and/or semantic similarity to labels of entities, the labels of the dimensions of entities in the entity graph, and/or a combination thereof. The interface aids may further enable a user to invoke display of the dimensions of an entity in the entity graph.
Each entity references identified at step 704 may be translated 708 into database language statements, e.g., SQL statements (“entity statements”). For example, if an entity represents a table 108, a database language statements referencing the table 108 may be generated. As noted above, a node in the knowledge graph may represent the result of a database operation 206 or operation 210 performed with respect to one or more tables 108 or the result of another operation 206 or operation 210. Accordingly, a database language statement may be generated that invokes execution one or more operations 206 or operations 210 with respect to one or more tables 108 represented by an entity referenced in the KGL statement.
The relationships identified at step 706 may be translated 710 into database language statements implementing an operation 206 and/or operation 210 corresponding to the recited relationships (“relationship statement”).
The method 700 may then include generating 712 a database query. For example, the entity statements and relationship statements may be combined along with other statements required by the database language to form a query representing the KGL statement. An entity may represent an operation performed with respect one or more objects that may be tables, the results of other operations, or the like. Accordingly, generating 712 the database query may include substituting database language statements to implement the operation represented by the entity and any operations generating a result processed by the operation.
The query may then be input to the server system 102 and a result of the query returned to the source of the KGL statement. The result of the query may be formatted or otherwise processed to facilitate understanding of the result, such as using Tableau or other interface generator.
Referring to
For example, the system 800 may receive a natural language statement 802 from a user, such as from an interface executing on the user computing device 116. The system 800 processes the natural language statement 802 using an LLM 804 to generate a KGL statement 806. The KGL statement 806 is a preliminary KGL statement and the entities referenced therein may not actually correspond to the labels of entities in the entity graph. However, the KGL statement 806 may have the form and structure of a KGL statement.
The LLM 804 may be trained with a plurality of KGL statements and instructed to imitate the structure and form of the KGL statements. The LLM may be general purpose LLM that is provide the KGL statements and the natural language statement 802 and instructed to convert the natural language statement 802 into a KGL statement.
The KGL statement may be processed using a semantic search module 808. The semantic search module 808 receives the KGL statement 806 and, for each entity referenced in the KGL statement 806 (“input entity”), searches for a corresponding entity in a knowledge graph 810 (“KG entity”).
The semantic search module 808 may access the knowledge graph 810 as well as semantic data 812. The semantic data 812 may include such information as concepts represented by words or phrases and relationships between concepts in the form of a graph. The labels of the entities in the knowledge graph 810 may be selected from among the concepts in the semantic data 812. The semantic module 808 may search for a concept in the knowledge graph 810 that is related to the input entity based on one or both of lexical (i.e., spelling) similarity and semantic (i.e., meaning) similarity as indicated in the semantic data 812. The semantic module 808 may, for example, seek for concepts related to each input entity in the semantic graph and attempt to identify the KG entity in the knowledge graph 810 that is connected to the same concepts either directly or by way of the labels for the dimensions of the entity.
The semantic search module 808 may further take into account the structure of the KGL statement 806. For example, where the KGL statement 806 includes reference A in the role of an entity, reference B as a dimension of the entity, and reference C as a value for the dimension, the semantic search module 808 will therefore search for an entity corresponding to reference A in terms of lexical and semantic similarity and having a dimension that corresponds to reference B in terms of lexical and semantic similarity, and having a possible value of C for the dimension. For example, in the KGL statement “Customer where Location=‘TX’ AND Birth Year=“1996”, the entity is Customer, the dimensions are Location and Birth Year, and the values are “TX” and “1996.” Where there are multiple dimensions, the dimensions may be searched separately, e.g. a combination of “Customer” and “Location” and a combination of “Customer and “Birth Year.” Entities identified as a result of both searches may then be evaluated to select an entity for “Customer” and dimensions of the entity corresponding to “Location” and “Birth Year.”
The semantic search module 808 replaces each input entity in the KGL statement with the corresponding KG entity identified by the semantic search module 808, where the KG entity is different from the input entity, to obtain KGL statement 814. The KGL statement may be displayed in an editor 816, such as using an interface displayed on the user device 116 from which the natural language statement 802 was received. The user may revise the KGL statement 814 using the editor 816. The user may revise the KGL statement 814 by changing the entities referenced in the KGL statement 814 and relationships between entities in the KGL statement 814. The editor 816 may implement some or all of the functions described above for an interface for receiving the input of KGL statements. In particular, the editor 816 may enable a user to view the knowledge graph, view the dimensions of an entity referenced in a KGL statement, provide suggested or possible values for an entity referenced in a KGL statement, or other aids to the editing of KGL statements.
If the user makes changes to the KGL statement 814, the revised KGL statement may be processed in various ways. In a first approach, the revised KGL statement is treated as a new natural language statement and processing repeats by inputting the revised KGL statement to the LLM 804. In a second approach, the revised KGL statement is input to the semantic search module to ensure that each entity referenced in the revised KGL statement is represented in the knowledge graph. In some embodiments, the first approach is used if the relationships included in the KGL statement 814 are revised and the second approach is used if only the entities referenced in the KGL statement 814 is changed.
Once a KGL statement 814 is approved by a user, the KGL statement 814 may be passed to a KGL interpreter 818. The KGL interpreter 818 interprets the KGL statement 814 to generate a database query 820. For example, the KGL interpreter 818 may implement the method 700 with respect to the KGL statement 814.
The database query 820 may be submitted to the server system 102, processed by the server system 102, and a result of the query returned to the source of the natural language statement 802, such as the user computing device 116. The result of the query 820 may be formatted or otherwise processed to facilitate understanding of the result, such as using Tableau or other interface generator.
As an example use case of the system 800, the natural language statement received from a user may be “customers who live in TX.” The KGL statement 814 derived therefrom may be “Customers where CurrentAddressState=‘TX.’” The database query 820 generated from the KGL statement 814 may be an SQL statement such as: WITH table_0 AD (SELECT customer.c_customer_sk AD output FROM tpcds_1000.customer JOIN tpcds_1000.customer_address AD customer_address_0 ON customer.c_current_addr_sk=customer_address_0.ca_address_sk WHERE customer_address_0.ca_state=‘TX’) SELECT*FROM table_0
As is readily apparent, generating the database query 820 would require much greater knowledge of SQL as well as the underlying databases 104. However, using the system 800, a user may generate the query 820 simply by typing a natural language statement and possibly editing a readily understandable KGL statement 814.
In some embodiments, the LLM may output an SQL statement 822 based on the natural language statement 802 rather than a KGL statement 806. In such embodiments, the SQL statement 822 may be converted to the KGL statement 806 that is then processed as described above. For example, the KGL interpreter 818 may be used to convert the SQL statement 822 into a KGL statement 806.
For example, a natural language statement 902 may be received from a user. The natural language statement 902 may be a natural language statement according to any of the examples described above. The natural language statement 902 may be processed to obtain a SQL statement 904 that is a conversion of the natural language statement 902 to an SQL statement that attempts to implement an intent indicated in the natural language statement 902. The SQL statement 904 may be obtained using the system 800 (e.g., the database query 820). The SQL statement 904 may be obtained from an LLM directly or with other logic preceding and/or following the LLM.
The SQL statement 904 may be processed using a repair stage 906. The repair stage 906 may operate in conjunction with a correction database 908. The operation of the repair stage 906 is described in detail below with respect to
The SQL statement 910 may then be executed with respect to a database, e.g., as a database query to obtain a result. The result may be returned to a source of the natural language statement 902, such as to a user device 116 from which the natural language statement 902 was received.
Each entry may include a final SQL statement 1006. The final SQL statement 1006 may be different from the original SQL statement 1002. The final SQL statement 1006 may also be the same as the original SQL statement 1002 or be replaced with an indicator that indicates that no correction to the original SQL statement 1002 was made. The final SQL statement 1006 may be the result of review by a human operator or by comparison to a human-generated SQL statement. For example, the original SQL statement 1002 may be generated based on a natural language statement describing an existing human-generated SQL statement. The final SQL statement 1006 may be the human-generated SQL statement.
Referring to
For example, there may be a set of entries 1000 including different natural language (NL) statements 1004a, 1004b, 1004c, respectively. Each of the entries 1000 in the set may include a correction 1100a, 1100b, 1100c, respectively. Each correction 1100a, 1100b, 1100c may be a change between the original SQL statement 1002 and the final SQL statement 1004 of the corresponding entry 1000. A correction 1100a, 1100b, 1100c may include changing a variable name, changing an SQL command, changing ordering of arguments, or any other change. Each correction 1100a, 1100b, 1100c may be in the form of an original text and a final text, where the original text is the text from the original SQL statement 1002 and the final text is text from the final SQL statement 1004. Each correction 1100a, 1100b, 1100c may be in the form of a regular expression, pattern, executable code, or other statement that defines how to transform an original text to a final text.
As shown in
The method 1200 may include searching 1206 for one or more matches for the subject natural language statement among the natural language statements 1004 of the entries 1000 in the correction database 908. Searching for matches may include using any metric of textual similarity (string edit distance, Jaccard index, etc.) and/or any search algorithm known in the art for identifying matches for an input query. The result of step 1206 may be a set of ranked results or a highest-ranked result according to the metric of textual similarity. Each result may be the natural language statement 1004 of an entry 1000 from the correction database 908 determined to be relevant to the subject natural language statement. The result of step 1206 may include the confidence score 912. For example, the metric of textual similarity for each result may be used as the confidence score 912 for the result. Alternatively, a value may be assigned by the search algorithm to indicate the relevance of a result and that value may be used as the confidence score 912.
The method 1200 may include making 1208 a correction decision according to the correction database 908 and implementing the correction decision in the cast that correction is needed. For example, if no match was found, the correction decision may include making no change to the subject SQL statement. If a match is found in which the original SQL statement and final SQL statement are the same, the correction decision may include making no change to the subject SQL statement. If a match is found that includes one or more corrections, the correction decision may include changing one or more parts of the subject SQL statement according to changes indicated in the matching entry 1000. For example, a correction indicated by the final SQL statement 1006 may be made to the subject SQL statement. Where the matching entry 1000 includes multiple corrections, some or all of the corrections may be implemented, e.g., those for which the original text is found in the subject SQL statement. Implementing a correction may include making a partial correction: text matching an original text of a correction may be replaced with the final text of the correction where some of the subject SQL statement is not part of the original text. Implementing a correction may include executing a regular expression, pattern, or executable code with respect to a portion of the subject SQL statement matching the original text (exactly or matching a regular expression language statement).
In some embodiments, the method may end at step 1208. The subject SQL statement including any modification made at step 1208 (“the output SQL statement”) may then be further processed. For example, the output SQL statement may be processed with respect to a database, transmitted to a user device 116 from which the subject natural language statement was received at step 1202, or stored for later use. Where the output SQL statement is transmitted to a user device, the confidence score 912 from step 1206 may be output along with the output SQL statement.
In other embodiments, the method 1200 may include generating 1210 a natural language statement based on the output SQL statement (“the output natural language statement”). For example, the output SQL statement may be input to an LLM as part of a prompt asking the LLM to generate a natural language statement summarizing the function invoked by the output SQL statement. The prompt may include one or more examples including example SQL statements and corresponding example natural language statements to guide the LLM in executing the prompt.
The output natural language statement output by the LLM in response to the prompt may be output 1212 to a user, e.g., transmitted to the user device 116 from which the original from which the subject natural language statement was received at step 1202. For example, the output natural language statement may be transmitted to the user computing device 116 of the user from which the subject natural language statement was received, such as to a web browser or client application on the user computing device 116. The output natural language statement may be output with the confidence score. The output natural language statement may facilitate an assessment by the user as to whether the output SQL statement expresses the intent of the user expressed in the subject natural language statement.
Referring to
The method may include receiving 1302 a natural language statement (“the original natural language statement”) and generating 1304 a KGL statement from the original statement (“the original KGL statement”), such as using the approach described above.
The method 1300 may further include generating 1306 one or more additional KGL statements based on the original KGL statement and connections in the knowledge graph. The additional KGL statements may be generated 1306 using some or all of the following approaches:
The method 1300 may include converting 1308 the additional KGL statements to SQL statements (“the additional SQL statements”). As noted above, KGL statements are directly and programmatically translatable into SQL statements. The additional SQL statements may be input 1310 to an LLM as part of a prompt instructing the LLM to generate an additional natural language statement for each of the additional SQL statement, the additional natural language statement summarizing the function performed by the additional SQL statement.
The additional natural language statements may be received 1312 from the LLM and output 1314. For example, the additional natural language statements may be output to the user as a suggestion. The additional natural language statements may be transmitted to the user computing device 116 from which the original natural language statement was received at step 1302. For example, the additional natural language statements may be transmitted to, and displayed by, a browser or client application from which the original natural language statement was received.
Each of the plurality of SQL statements may be executed 1406 to obtain a result such that a plurality of results are obtained from executing the plurality of SQL statements. The result may be one or more tables of values. The tables may be labeled and may have column labels and possibly row labels.
The method 1400 may include inputting 1408 a prompt LLM to generate a report from the plurality of results along with the plurality of results. The prompt may include examples, each example including example results and an example report. Each example report may be a human generated report based on the example results.
A report may be received 1410 from the LLM in response to the prompt and the report may be output 1412. For example, the report may be transmitted to the user computing device 116 from which the original natural language statement was received at step 1402. For example, the report may be transmitted to, and displayed by, a browser or client application from which the original natural language statement was received.
Computing device 1500 includes one or more processor(s) 1502, one or more memory device(s) 1504, one or more interface(s) 1506, one or more mass storage device(s) 1508, one or more Input/Output (I/O) device(s) 1510, and a display device 1530 all of which are coupled to a bus 1512. Processor(s) 1502 include one or more processors or controllers that execute instructions stored in memory device(s) 1504 and/or mass storage device(s) 1508. Processor(s) 1502 may also include various types of computer-readable media, such as cache memory. The processor 1502 may be embodied as or further include a graphics processing unit (GPU) including multiple processing cores.
Memory device(s) 1504 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 1514) and/or nonvolatile memory (e.g., read-only memory (ROM) 1516). Memory device(s) 1504 may also include rewritable ROM, such as Flash memory.
Mass storage device(s) 1508 include various computer readable media, such as magnetic tapes, magnetic disks, optical disks, solid-state memory (e.g., Flash memory), and so forth. As shown in
I/O device(s) 1510 include various devices that allow data and/or other information to be input to or retrieved from computing device 1500. Example I/O device(s) 1510 include cursor control devices, keyboards, keypads, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, lenses, CCDs or other image capture devices, and the like.
Display device 1530 includes any type of device capable of displaying information to one or more users of computing device 1500. Examples of display device 1530 include a monitor, display terminal, video projection device, and the like.
Interface(s) 1506 include various interfaces that allow computing device 1500 to interact with other systems, devices, or computing environments. Example interface(s) 1506 include any number of different network interfaces 1520, such as interfaces to local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. Other interface(s) include user interface 1518 and peripheral device interface 1522. The interface(s) 1506 may also include one or more peripheral interfaces such as interfaces for printers, pointing devices (mice, track pad, etc.), keyboards, and the like.
Bus 1512 allows processor(s) 1502, memory device(s) 1504, interface(s) 1506, mass storage device(s) 1508, I/O device(s) 1510, and display device 1530 to communicate with one another, as well as other devices or components coupled to bus 1512. Bus 1512 represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.
For purposes of illustration, programs and other executable program components are shown herein as discrete blocks, although it is understood that such programs and components may reside at various times in different storage components of computing device 1500, and are executed by processor(s) 1502. Alternatively, the systems and procedures described herein can be implemented in hardware, or a combination of hardware, software, and/or firmware. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein.
In the above disclosure, reference has been made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific implementations in which the disclosure may be practiced. It is understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
Implementations of the systems, devices, and methods disclosed herein may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed herein. Implementations within the scope of the present disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, implementations of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media (devices) and transmission media.
Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.
Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, an in-dash vehicle computer, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
Further, where appropriate, functions described herein can be performed in one or more of: hardware, software, firmware, digital components, or analog components. For example, one or more application specific integrated circuits (ASICs) can be programmed to carry out one or more of the systems and procedures described herein. Certain terms are used throughout the description and claims to refer to particular system components. As one skilled in the art will appreciate, components may be referred to by different names. This document does not intend to distinguish between components that differ in name, but not function.
These example devices are provided herein purposes of illustration, and are not intended to be limiting. Embodiments of the present disclosure may be implemented in further types of devices, as would be known to persons skilled in the relevant art(s). At least some embodiments of the disclosure have been directed to computer program products comprising such logic (e.g., in the form of software) stored on any computer useable medium. Such software, when executed in one or more data processing devices, causes a device to operate as described herein.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on a computer system as a stand-alone software package, on a stand-alone hardware unit, partly on a remote computer spaced some distance from the computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
The present invention is described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions or code. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a non-transitory computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. The foregoing description has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate implementations may be used in any combination desired to form additional hybrid implementations of the disclosure.
This application is a continuation-in-part of U.S. patent application Ser. No. 18/188,059 filed Mar. 22, 2023, entitled KNOWLEDGE GRAPH GENERATION FOR A DATA WAREHOUSE, which claims the benefit of U.S. Provisional Patent Application 63/484,302 filed Feb. 10, 2023, entitled AUTOMATIC KNOWLEDGE GRAPH GENERATION AND AUTOMATIC QUERY GENERATION USING A KNOWLEDGE GRAPH AND KNOWLEDGE GRAPH LANGUAGE, and this application is a continuation-in-part of U.S. patent application Ser. No. 18/188,083 filed Mar. 22, 2023, entitled DATABASE QUERY GENERATION FROM NATURAL LANGAUGE STATEMENTS, which claims the benefit of U.S. Provisional Patent Application 63/484,302 filed Feb. 10, 2023, entitled AUTOMATIC KNOWLEDGE GRAPH GENERATION AND AUTOMATIC QUERY GENERATION USING A KNOWLEDGE GRAPH AND KNOWLEDGE GRAPH LANGUAGE. All of the foregoing are hereby incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63484302 | Feb 2023 | US | |
| 63484302 | Feb 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18188059 | Mar 2023 | US |
| Child | 18887526 | US | |
| Parent | 18188083 | Mar 2023 | US |
| Child | 18887526 | US |
