Patent Application
20250124308
The devices and methods disclosed in this document relate to data visualization and, more particularly, to interactive visualization of large language model design knowledge.
Unless otherwise indicated herein, the materials described in this section are not admitted to be the prior art by inclusion in this section.
Idea generation is a pivotal phase in the design process, introducing concepts for subsequent development, modeling, and manufacturing. Two common challenges in achieving high-quality design ideation are: first, exploring the design space in a breadth-first manner to generate as many design concepts as possible, and second, delving into the potential of these concepts in depth to fully uncover their capabilities. To overcome these challenges, textual stimuli have been widely used to provide designers with concepts and information related to their original thinking. Textual stimuli can involve design datasets consisting of existing designs and ideas to query and retrieve design information for inspiration. Knowledge graph-based datasets have been created to fulfill this requirement.
Recent advances in Generative Artificial Intelligence (GAI) and Large Language Models (LLMs) have opened up new possibilities to enhance human design capabilities by integrating diverse knowledge elements into cohesive patterns. This allows for critical thinking and learning through the formation of concept connections and divergent investigations. LLMs are a type of GAI that generates textual responses based on given input contexts. They are commonly used for language-related tasks, such as translation, answering questions, and text comprehension. Recently, LLMs have gained attention from the design community for their potential to support text stimuli in design ideation. Unlike previous creative design tools that retrieve information from explicit knowledge datasets, LLMs can infer implicit information from generative pre-training and possess contextual understanding, enabling them to generate creative and context-relevant responses for multiple design tasks.
However, the challenge of effectively managing and leveraging the overwhelming volume of textual responses from LLMs can hinder efficient collaboration with LLMs in design ideation. To improve this collaboration, a system is needed to address the current inefficiencies in the interaction between humans and LLMs during the design process.
A method for visualizing natural language response of a language model is disclosed. The method comprises displaying, on a display, a graphical user interface. The method further comprises generating, with a processor, a first natural language prompt based on first user inputs received via the graphical user interface. The method further comprises receiving, with the processor, a first natural language response from the language model that is responsive to the first natural language prompt. The method further comprises generating, with the processor, a knowledge graph representing the first natural language response. The method further comprises displaying, on the display, a graphical representation of the knowledge graph in the graphical user interface.
The foregoing aspects and other features of the system and method are explained in the following description, taken in connection with the accompanying drawings.
For the purposes of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It is understood that no limitation to the scope of the disclosure is thereby intended. It is further understood that the present disclosure includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles of the disclosure as would normally occur to one skilled in the art to which this disclosure pertains.
Large Language Models (LLMs) open up myriad possibilities for augmenting the idea-generation processes of human designers. However, how human designers can best work with such models is less well understood, and the ideas generated by LLMs are often found to be redundant and fragmented. To address this, an interactive knowledge visualization system 100 is introduced that enables structured human-AI collaborative design ideation. Particularly, the interactive knowledge visualization system 100 advantageously organizes LLM responses into an interactive knowledge graph visualization. The interactive knowledge visualization system 100 advantageously provides a well-organized framework that empowers users to manage the knowledge provided by the LLM, with the aid of a visual interface. Moreover, the interactive knowledge visualization system 100 mitigates design fixation and enhances the overall efficiency, quality, quantity, and depth of concepts in the ideation process with the aid of this symbiotic human-LLM visual interface. Thus, the interactive knowledge visualization system 100 enhances overall efficiency and quality in the ideation process.
The interactive knowledge visualization system 100 provides graphical user interfaces to facilitate efficient exploration of LLM-generated knowledge through interactive visualization. Thus, the graphical user interface supports human-AI collaborative ideation through the interactive visualization of LLM-generated design knowledge.
With reference to
Based on the text inputs and option selections received from the user via the sidebar menu, the interactive knowledge visualization system 100 generates a natural language prompt for an LLM 150 to generate a response that includes knowledge regarding the provided keywords or key phrases. The interactive knowledge visualization system 100 generates an initial suggestion of a natural language prompt based on the keywords or key phrases input by the user and any other text inputs or selected options in the sidebar menu. The interactive knowledge visualization system 100 incorporates a comprehensive prompt library. The prompt library includes prompts associated with specific design activities, including concept generation, exploration, comparison, and critiquing. Additionally, the prompt library includes prompts associated with specific design methods, including SCAMPER (“Substitute, Combine, Adjust, Modify, Put to other uses, Eliminate, Reverse”), brainstorming, and functional decomposition.
After the interactive knowledge visualization system 100 generates a suggested natural language prompt, the user can edit and revise the prompt until they are satisfied. With reference to
Once the user has finalized their initial natural language prompt, the prompt is provided to the LLM 150 to generate a natural language response. The interactive knowledge visualization system 100 processes the natural language response to generate a knowledge graph representing the information contained therein. Moreover, the interactive knowledge visualization system 100 displays an interactive visualization of the knowledge graph within a knowledge graph canvas of the graphical user interface.
This knowledge graph structure serves as a visual representation of the design knowledge collaboratively generated by both the human designer and the LLM 150. In some embodiments, the knowledge graph is structured around three distinct types of nodes, including concept nodes, information nodes, and example nodes, thereby enhancing visualization, understanding, and interaction. The generated knowledge graph helps users initiate subsequent ideation iterations, leading to the refinement of concepts until the most preferred concept is obtained. Additionally, structuring the design concepts with a knowledge graph prevents users from being overwhelmed by extensive LLM responses.
With reference to
The interactive knowledge visualization system 100 supports direct interaction with the knowledge graph through an interaction menu. With reference to
The backend server 110 may include one or more servers configured to serve a variety of functions for the interactive knowledge visualization system 100, including web servers or application servers depending on the features provided by the interactive knowledge visualization system 100. Each backend server 110 includes, for example, a processor 112, a memory 114, and a network communications module 116. It will be appreciated that the illustrated embodiment of the backend server 110 is only one exemplary embodiment of a backend server 110 and is merely representative of any of various manners or configurations of a personal computer, server, or any other data processing system that is operative in the manner set forth herein.
The processor 112 is configured to execute instructions to operate the backend server 110 to enable the features, functionality, characteristics and/or the like as described herein. To this end, the processor 112 is operably connected to the memory 114 and the network communications module 116. The processor 112 generally comprises one or more processors which may operate in parallel or otherwise in concert with one another. It will be recognized by those of ordinary skill in the art that a “processor” includes any hardware system, hardware mechanism or hardware component that processes data, signals or other information. Accordingly, the processor 112 may include a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuitry for achieving functionality, programmable logic, or other processing systems.
The memory 114 is configured to store program instructions that, when executed by the processor 112, enable the backend server 110 to perform various operations described herein. The memory 114 may be any type of device or combination of devices capable of storing information accessible by the processor 112, such as memory cards, ROM, RAM, hard drives, discs, flash memory, or any of various other computer-readable media recognized by those of ordinary skill in the art. The memory 114 stores an interactive knowledge visualization program 118, as well as prompt and rules libraries 120. The processor 112 executes program instructions of the interactive knowledge visualization program 118 to recommend natural language prompts to a user based on user inputs, using prompt templates of the prompt and rules libraries 120. Additionally, the processor 112 executes program instructions of the interactive knowledge visualization program 118 to generate knowledge graphs and visualizations of natural language responses provided by the LLM 150, using rulesets of the prompt and rules libraries 120.
The network communications module 116 may comprise one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in a communications module to enable communications with various other devices, at least including the client device 130. In particular, the network communications module 116 may include a local area network port that allows for communication with any of various local computers housed in the same or nearby facility. Generally, the backend server 110 communicates with remote computers over the Internet via a separate modem and/or router of the local area network. Alternatively, the network communications module 116 may further include a wide area network port that allows for communications over the Internet. In one embodiment, the network communications module 116 is equipped with a Wi-Fi transceiver or other wireless communications device. Accordingly, it will be appreciated that communications with the backend server 110 may occur via wired communications or via the wireless communications. Communications may be accomplished using any of various known communications protocols.
With continued reference to
The processor 132 is configured to execute instructions to operate the client device 130 to enable the features, functionality, characteristics and/or the like as described herein. To this end, the processor 132 is operably connected to the memory 134, the network communications module 136, and the display screen 138. The processor 132 generally comprises one or more processors which may operate in parallel or otherwise in concert with one another. It will be recognized by those of ordinary skill in the art that a “processor” includes any hardware system, hardware mechanism or hardware component that processes data, signals, or other information. Accordingly, the processor 132 may include a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuitry for achieving functionality, programmable logic, or other processing systems.
The memory 134 is configured to store data and program instructions that, when executed by the processor 132, enable the client device 130 to perform various operations described herein. The memory 134 may be any type of device capable of storing information accessible by the processor 132, such as a memory card, ROM, RAM, hard drives, discs, flash memory, or any of various other computer-readable media serving as data storage devices, as will be recognized by those of ordinary skill in the art.
The network communications module 136 may comprise one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in a communications module to enable communications with various other devices, at least including the backend server 110. Particularly, the network communications module 136 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and/or Wi-Fi router (not shown). Additionally, the network communications module 136 may include a Bluetooth® module (not shown) configured to enable communication with the backend server 110. Finally, the network communications module 136 may include one or more cellular modems configured to communicate with wireless telephony networks.
The display screen 138 may comprise any of various known types of displays, such as LCD or OLED screens. In some embodiments, the display screen 138 may comprise touch screens configured to receive touch inputs from a user. Alternatively, or in addition, the client device 130 may include additional user interfaces, such as buttons, switches, a keyboard or other keypad, speakers, and a microphone.
A variety of methods, workflows, and processes are described below for enabling the operations of the interactive knowledge visualization system 100. In these descriptions, statements that a method, workflow, processor, and/or system is performing some task or function refers to a controller or processor (e.g., the processor 112 of the backend server 110 or the processor 132 of the client device 130) executing programmed instructions (e.g., the interactive knowledge visualization program 118) stored in non-transitory computer readable storage media (e.g., the memory 114 of the backend server 110 or the memory 134 of the client device 130) operatively connected to the controller or processor to manipulate data or to operate one or more components in the interactive knowledge visualization system 100 to perform the task or function. Additionally, the steps of the methods may be performed in any feasible chronological order, regardless of the order shown in the figures or the order in which the steps are described.
The method 200 begins with displaying a graphical user interface to a user (block 210). Particularly, the processor 132 of the client device 130 operates the display screen 138 to display a graphical user interface. The graphical user interface may be that of a native application on the client device 130 or a web-based interface displayed in an internet browser application on the client device 130. In the native application embodiment, the client device 130 executes the native application to render the graphical user interface on the display screen 138. In the web-based embodiment, the client device 130 receives elements of the graphical user interface via the network communications module 136 and renders the graphical user interface within an Internet browser application on the display screen 138. In one embodiment, the graphical user interface is implemented using Streamlit, a Python-based framework for deploying machine learning applications on the web.
The method 200 continues with generating a natural language prompt based on user inputs received via the graphical user interface (block 220). Particularly, the processor 112 of the backend server and/or the processor 132 of the client device 130 generates a natural language prompt based on user inputs received via the graphical user interface. Firstly, the processor 112 and/or the processor 132 receives a plurality of user inputs from the user via user interactions with the graphical user interface. The plurality of user inputs at least includes text inputs and a selected option from a plurality of predefined options. Next, the processor 112 and/or the processor 132 matches the selected option to a corresponding natural language prompt template from a plurality of natural language prompt templates stored in a prompt library (i.e., in the prompt and rules libraries 120). Finally, the processor 112 and/or the processor 132 generates the natural language prompt by incorporating the text inputs into slots of the natural language prompt template. The processor 132 operates the display screen 138 to display the generated natural language prompt to the user in the graphical user interface.
The graphical user interface 400A further includes one or more text fields 404 for populating a natural language prompt template. In at least some embodiments, each of the natural language prompt templates in the prompt library corresponds to a particular option from the plurality of user-selectable options 402. Accordingly, in some embodiments, the processor 112 and/or the processor 132 automatically updates the particular text fields 404 depending on the user-selectable option 402 that has been selected. In the illustrated example, the user has selected the “Generate” option and the text fields 404 include two text fields relating to the design goals of the “Generate” design activity: “Please enter your keyword concept:” and “Please enter the customer requirements.” In the first text field 404, the user types text defining a topic of the natural language prompt (e.g., “Outdoor Toy”) and, in the second text field 404, the user types text defining constraints on the natural language response (e.g., constraints on the design goal, “Mechanically Interactive” and “Safe”).
The graphical user interface 400A further includes a generated prompt text field 406. Once the user has selected one of the user-selectable options 402 and entered text into the text fields 404, the processor 112 and/or the processor 132 generates a suggested natural language prompt (e.g., “Can you list more ideas related to the concept of an Outdoor Toy that is Mechanically Interactive and Safe?”). The natural language prompt is displayed within the generated prompt text field 406. In at least some embodiments, the generated prompt text field 406 is a user-editable text field and the processor 132 receives edits from the user via the graphical user interface 400A, thereby defining an edited natural language prompt.
In at least some embodiments, natural language prompt templates include multiple variations (not shown) that are used depending on whether the user provides text inputs for the various text fields provided in the graphical user interface. Particularly, depending on whether certain text fields are left blank, the processor 112 and/or the processor 132 selects a different variation of the natural language prompt template corresponding to the selected option. For example, the “Generate” template includes a variation that incorporates the variable slot ‘requirements’ which define constraints on the natural language response. The variable slot ‘requirements’ is populated by further text inputs from the user, e.g., in the “Please enter the customer requirements” text field 404 of
With continued reference to
The “Critique” template enables designers to drill deeper into a specific idea from the perspective of the design constraints to uncover advantages and disadvantages. The “Critique” template includes the variable slot ‘keyword_concept’ that defines the topic of critique, and which is populated by text inputs from the user.
The “Explore” template enables designers to creatively find implementation methods for a function/subfunction of a concept. The “Explore” template includes the variable slots ‘selected_function’ and ‘parent_concept’ that define the topic of exploration, and which are populated by text inputs from the user.
The “Functional Decomposition” template enables designers to focus on specific aspects of a design problem for more effective ideation. The “Functional Decomposition” template at least includes the variable slot ‘keyword_concept’ that defines the topic of the functional decomposition, and which is populated by text inputs from the user.
The “SCAMPER” template enables designers to apply the SCAMPER method, which encourages designers to think outside the box by considering existing designs from different perspectives by asking questions about how they can be modified or enhanced. The “SCAMPER” template includes the variable slot ‘keyword_concept’ that defines the topic of the SCAMPER method, and which is populated by text inputs from the user.
Finally, the “Brainstorm” template enables designers to think creatively by brainstorming further ideas. The “Brainstorm” template includes the variable slot ‘keyword_concept’ that defines the topic of the brainstorming, and which is populated by text inputs from the user.
Returning to
The language model is a machine learning-based model, for example in the form of an artificial neural network. The language model is configured to receive natural language text as an input prompt and generate natural language text as an output response. In at least some embodiments, the language model is a large language model (LLM) 150, such as OpenAI's ChatGPT™, Google's Gemini™, or Anthropic's Claude™. An LLM is a generative machine learning model that is trained on vast amounts of textual data to understand and generate human-like responses to natural language prompts. These models are designed to predict and produce coherent and contextually relevant text, imitating human language fluency. They work by analyzing patterns in language data, learning grammar, context, and meaning, and then using that knowledge to generate new content.
In general, the LLM 150 is implemented by a remote third-party server rather than being executed directly by the client device 130 or by the backend server 110. Instead, the interactive knowledge visualization system 100 interfaces with the LLM 150 via Internet communications using an API. Particularly, once the natural language prompt is finalized, the processor 112 operates the network communications module 116 and/or the processor 132 operates the network communications module 136 to transmit a message including the natural language prompt to a server hosting the LLM 150. In response, the processor 112 receives via the network communications module 116 and/or the processor 132 receives via the network communications module 136 a natural language response from the LLM 150 that includes text that is responsive to the natural language prompt. However, in alternative embodiments, the backend server 110 and/or the client device 130 stores the LLM 150 and executes the LLM 150 to generate the natural language response locally.
In at least some embodiments, the processor 112 and/or the processor 132 is configured to incorporate additional language to the generated natural language prompt prior to providing it to the LLM 150, for the purpose of prompt engineering. For example, the natural language prompt may be modified to incorporate the additional language: “Please generate the response as a list of key concepts, each including a corresponding description.” This additional language is hidden from the user (i.e., not displayed in the generated prompt text field 406) but is designed to instruct the LLM 150 to generate the natural language response in a particular structured format. In some embodiments, the particular structured format and the additional language included in the natural language prompt is different depending on the natural language prompt template that was used. As will be discussed below, the particular structured format will be parsed to extract information from the natural language response for the purpose of generating a knowledge graph.
In at least some embodiments, the backend server 110 and/or the client device 130, or the LLM 150 itself, implements a LangChain algorithm to preserve the memory and context of the discussion with the LLM 150. The LangChain algorithm streamlines interactions with the LLM 150 by allowing for more complex, multi-step tasks. The LangChain algorithm connects different components such as memory, data retrieval, and APIs, enabling the LLM 150 to work with structured workflows of the interactive knowledge visualization system 100.
The method 200 continues with generating a knowledge graph representing the natural language response (block 240). Particularly, the processor 112 of the backend server and/or the processor 132 of the client device 130 generates a knowledge graph representing the natural language response received from the LLM 150. As used herein, a “knowledge graph” refers to any structured representation of information that organizes data into nodes (entities) and edges (relationships) to capture relationships between concepts in the information. Thus, the processor 112 and/or the processor 132 generates the knowledge graph including a plurality of nodes and a plurality of edges that connect respective pairs of nodes in the plurality of nodes.
To generate the knowledge graph, the processor 112 and/or the processor 132 extracts a plurality of keywords from the natural language response and defines respective nodes of the knowledge graph for each keyword. Additionally, the processor 112 and/or the processor 132 extracts a respective keyword description for each respective keyword from the natural language response and associates the respective keyword description with the corresponding node in the knowledge graph. As discussed above, in some embodiments, the natural language prompt incorporates additional text that instructs the LLM 150 to generate the natural language response in a particular structured format. In such embodiments, the processor 112 and/or the processor 132 extracts the keywords and keyword descriptions by parsing the particular structured format of the natural language response according to a predefined ruleset selected from a plurality of predefined rulesets in a ruleset library (i.e., in the prompt and rules libraries 120). In some embodiments, the processor 112 and/or the processor 132 uses a different predefined ruleset depending on the particular natural language prompt template that was used to generate the natural language prompt. In this way, the keywords and keyword descriptions can be easily extracted from the natural language response. In alternative embodiments, the processor 112 and/or the processor 132 may utilize a machine learning-based keyword and keyword description extraction technique.
In some embodiments, the processor 112 and/or the processor 132 defines a central node of the knowledge graph for a user-provided keyword from the first natural language prompt (e.g., the text input “Outdoor Toys” in the example of
In some embodiments, the processor 112 and/or the processor 132 classifies each node of the knowledge graph as a respective node type from a plurality of node types (e.g., “Concept” nodes, “Information” nodes, and “Example” nodes). As discussed in greater detail below, each node of the knowledge graph is displayed in a visually distinctive manner that identifies the node type. In some embodiments, the processor 112 and/or the processor 132 classifies each node of the knowledge graph depending on the natural language prompt template that was used to generate the natural language prompt from which the natural language response was generated. For example, if the natural language response was generated based on a “Generate” prompt, then the nodes defined based on keywords in the natural language response are classified as “Concept” nodes. As another example, if the natural language response was generated based on an “Explore” prompt, then the nodes defined based on keywords in the natural language response are classified as “Example” nodes. As a final example, if the natural language response was generated based on a “Compare” prompt or “Critique” prompt, then the nodes defined based on keywords in the natural language response are classified as “Information” nodes.
In some embodiments, the processor 112 and/or the processor 132 retrieves a representative image for each keyword and associates the representative image with the corresponding node in the knowledge graph. Particularly, for each node in the knowledge graph, the processor 112 and/or the processor 132 retrieves a representative image using the image search platform 160, e.g., Bing image search or Google image search. The image search may be performed based on the keyword and/or the keyword description associated with the respective node. Once a representative image is retrieved, the processor 112 and/or the processor 132 associates the representative image with the respective node in the knowledge graph.
Returning to
The knowledge graph can be visualized in a wide variety of ways. In at least one embodiment, the processor 112 and/or the processor 132 generates the visualization of the knowledge graph as a node-link diagram, in which entities are represented as nodes and relationships are represented as edges connecting the nodes in a web-like manner. In another embodiment, the processor 112 and/or the processor 132 generates the visualization of the knowledge graph as a hierarchical tree that helps to visualize parent-child relationships between nodes of the knowledge graph. Such hierarchical trees can be flattened into a hierarchical linear list for easy viewing and navigation by simply scrolling up and down through the hierarchical linear list. In another embodiment, the processor 112 and/or the processor 132 generates the visualization of the knowledge graph as an adjacency list and/or an adjacency matrix, which displays relationships between entities in a grid-like format, offering a more compact representation of the knowledge graph. It should be appreciated that these are only a few of the wide variety of techniques that might be used to visualize the knowledge graph.
In the illustrated embodiment, each node 420, 422A-F in the knowledge graph is displayed in association with the previously retrieved representative image associated with the node. Particularly, in one embodiment, to make it easier for the user to quickly understand what each node represents, the processor 112 and/or the processor 132 renders each of the nodes 420, 422A-F with the representative image depicted within the circle and with a text label beneath the circle.
In addition to the graphical depiction of the knowledge graph, the graphical user interface of the interactive knowledge visualization system 100 further enables the user to directly explore the text of the natural language response provided by the LLM 150. To this end, the processor 132 operates the display screen 138 to display one or more previously received natural language responses from the LLM 150 in the graphical user interface. Additionally, the processor 132 operates the display screen 138 to display one or more previously provided user inputs that were used to generate previously generated natural language prompts.
The method 300 begins with generating a further natural language prompt based on further user inputs received via the graphical user interface (block 310). Particularly, the interactive knowledge visualization system 100 enables users to expand the knowledge graph to include additional information extracted from further natural language responses with respect to particular nodes in the knowledge graph. To this end, the processor 112 of the backend server and/or the processor 132 of the client device 130 generates a further natural language prompt based on further user inputs received via the graphical user interface.
In an essentially similar manner as discussed above, the processor 112 and/or the processor 132 receives a plurality of user inputs from the user via user interactions with the graphical user interface. The plurality of user inputs includes a selected option from a plurality of predefined options. However, in contrast to generating the initial natural language prompt, the plurality of user inputs also includes a selection of a selected node from the plurality of nodes of the knowledge graph. Next, the processor 112 and/or the processor 132 matches the selected option to a corresponding natural language prompt template from a plurality of natural language prompt templates stored in the prompt library (i.e., in the prompt and rules libraries 120). Finally, the processor 112 and/or the processor 132 generates the natural language prompt by incorporating the keyword and/or keyword description associated with the selected node into the natural language prompt template. The processor 132 operates the display screen 138 to display the generated natural language prompt to the user in a graphical user interface.
Upon the user selection of a node in the visualization 440, an interaction menu 450 will display on the right side of the graphical user interface 400C. The interaction menu 450 includes a drop-down menu 452 from which the user can select from a plurality of user-selectable options. Several of the user-selectable options in the drop-down menu 452 correspond to particular natural language prompt template in the prompt library. In some embodiments, the drop-down menu 452 also includes additional options for managing the knowledge graph, such as options for deleting nodes, deleting edges, manually connecting nodes, or annotating/marking nodes. In the illustrated example, the concept node 422F in the visualization 440 (i.e., “Gardening tools”) was selected by the user and the user has selected the “Explore” option from the drop-down menu 452. Additionally, the interaction menu 450 includes a keyword description 454 associated with the selected node.
After the user has selected a node and selected an option from the drop-down menu 452, the processor 112 and/or the processor 132 matches the selected option to a corresponding natural language prompt template from the prompt library. The processor 112 and/or the processor 132 generates the natural language prompt by incorporating the keyword and/or keyword description associated with the selected node into the natural language prompt template. The processor 132 operates the display screen 138 to display the generated natural language prompt to the user within the generated prompt text field 406 of the sidebar menu of
It should be appreciated that, in some embodiments, a different variation of the natural language prompt template may be used for prompts generated using interaction menu 450 compared to the initial prompt generation using the sidebar menu of
The method 300 continues with receiving a further natural language response from a language model that is responsive to the further natural language prompt (block 320). Particularly, in an essentially similar manner that was discussed previously with respect to block 230 of the method 200, the processor 112 of the backend server and/or the processor 132 of the client device 130 provides the generated, and optionally edited, natural language prompt to the LLM 150 for processing. Based on the natural language prompt, the processor 112 and/or the processor 132 receives a natural language response from the language model that is responsive to the provided natural language prompt. With reference again to
The method 300 continues with expanding the knowledge graph to further represent the further natural language response (block 330). Particularly, the processor 112 of the backend server and/or the processor 132 of the client device 130 expands the knowledge graph to further represent the further natural language response received from the LLM 150. Specifically, the processor 112 and/or the processor 132 expands the knowledge graph by adding additional nodes and/or additional edges to the knowledge graph based on the further natural language response.
In an essentially similar manner as was discussed with respect to block 240 of the method 200, to expand the knowledge graph, the processor 112 and/or the processor 132 extracts a plurality of keywords from the further natural language response and defines respective nodes of the knowledge graph for each keyword. Additionally, the processor 112 and/or the processor 132 extracts a respective keyword description for each respective keyword from the further natural language response and associates the respective keyword description with the corresponding node in the knowledge graph.
In some embodiments, the processor 112 and/or the processor 132 defines a respective edge in the knowledge graph connecting each of the newly added nodes for the keywords in the further natural language response to the previously selected node in the knowledge graph that was used to generate the further natural language prompt. For example, in the illustrated example of
In some embodiments, the processor 112 and/or the processor 132 classifies each new node of the knowledge graph as a respective node type from a plurality of node types (e.g., “Concept” nodes, “Information” nodes, and “Example” nodes) depending on the natural language prompt template that was used to generate the natural language prompt from which the natural language response was generated. For example, in the illustrated example of
The method 300 continues with displaying a graphical representation of the expanded knowledge graph in the graphical user interface (block 340). Particularly, the processor 112 and/or the processor 132 generates a graphical representation of the expanded knowledge graph. The processor 132 of the client device 130 operates the display screen 138 to display the graphical representation of the expanded knowledge graph in the graphical user interface.
With reference again to
In a similar manner, the LLM 150 was further prompted to perform functional decomposition with respect to the “Gardening gloves” example node 424D using the “Functional Decomposition” prompt. The knowledge graph was subsequently expanded based on the natural language response to the “Functional Decomposition” prompt. As a result of this expansion, the plurality of nodes further includes information nodes 426A-E. The information nodes 426A-E are nodes of the “Information” node type and include information/keywords extracted from the natural language response (e.g., “Safety,” “Versatility,” “Ergonomics,” “Compatibility,” and “Accessibility”). Each of the information nodes 426A-E are connected by edges to the “Gardening gloves” example node 424D that was selected to generate the “Functional Decomposition” prompt.
In the illustrated embodiment, each node 420, 422A-F, 424A-F, 426A-E in the knowledge graph is displayed in a visually distinctive manner that identifies a node type (e.g., “Concept” nodes, “Information” nodes, and “Example” nodes). Particularly, in one embodiment, the processor 112 and/or the processor 132 displays the concept nodes 422A-F with a first color (e.g., red), displays the example nodes with a second color 424A-F (e.g., blue), and displays the information nodes 426A-E with a third color (e.g., green).
In the illustrated embodiment, each node 420, 422A-F, 424A-F, 426A-E in the knowledge graph is displayed in association with the previously retrieved representative image associated with the node. Particularly, in one embodiment, to make it easier for the user to quickly understand what each node represents, the processor 112 and/or the processor 132 renders each of the nodes 420, 422A-F, 424A-F, 426A-E with the representative image depicted within the circle and with a text label beneath the circle.
Design ideation begins with the generation of new design concepts. Typically, a designer is provided with a usage scenario for the target design, along with specific design constraints that outline customer requirements such as safety, cost, and recommended age grading. The interactive knowledge visualization system 100 aids the designer in thinking outside the box and developing distinct perspectives on the given problem to identify needs and features.
Through visual interactions and automatic knowledge graph expansion, users can seamlessly organize ideas and track iterations of their concepts without interrupting their creative flow. For instance, in illustration (1) of
Subsequently, the user prompts the LLM 150 using suggested prompts, as depicted in illustration (2) of
In some situations, designers are required to refine an existing design to enhance its performance or adapt it to different application contexts. To do this effectively, the designer must acquire knowledge about the various features of the design and understand their limitations. An example of this process is illustrated in
Comparing design concepts is an efficient way to build knowledge about the problem and stimulate human creativity for novel ideas. The interactive knowledge visualization system 100 empowers designers with knowledge from LLMs to comprehensively and efficiently compare design concepts from distinct perspectives. In illustration (1) of
As shown in illustration (2) of
Embodiments within the scope of the disclosure may also include non-transitory computer-readable storage media or machine-readable medium for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored thereon. Such non-transitory computer-readable storage media or machine-readable medium may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such non-transitory computer-readable storage media or machine-readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. Combinations of the above should also be included within the scope of the non-transitory computer-readable storage media or machine-readable medium.
Computer-executable instructions include, for example, instructions and data which cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
While the disclosure has been illustrated and described in detail in the drawings and foregoing description, the same should be considered as illustrative and not restrictive in character. It is understood that only the preferred embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.
This application claims the benefit of priority of U.S. provisional application Ser. No. 63/543,933, filed on Oct. 13, 2023 the disclosure of which is herein incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63543933 | Oct 2023 | US |
