Patent Application
20250097272
Online communications platforms have become an increasingly important tool to facilitate collaboration among meeting participants. The communication platforms provide audio and video content of the participants but can also provide the ability for users to share content with other users. This content can include non-interactive content that is presented by a participant via screen sharing or interactive content that enables the participants to interactive with and/or collaborate on the creation of the interactive content. In particular, the inclusion of interactive content in an online meeting can increase participant engagement, retention of information and shared understanding. However, many meeting participants are often unaware of the ability to add such interactive content to an online meeting, unaware how to add such interactive content to the online meeting, or are aware that interactive content exists, but find it onerous to set up. Hence, there is a need for improved systems and methods that provide means for automatically assisting meeting participants in the creation of interactive content during an online meeting.
An example data processing system according to the disclosure includes a processor and a memory storing executable instructions. The instructions when executed cause the processor alone or in combination with other processors to perform operations including detecting an occurrence of a trigger condition during an online communication session among a plurality of client devices associated with participants of the online communication session, the occurrence of the trigger condition indicating that a visualization of content associated with the online communication session should be generated and presented on the client devices associated with the participants of the online communication session; selecting a candidate visualization layout from a plurality of visualization layouts in a visualization layout datastore based on the meeting category associated with the online communication session; constructing a prompt for a language model based on the selected candidate visualization layout and a transcript of the online communication session; providing the prompt as an input to a language model; obtaining visualization information as an output from the language model, wherein the visualization information is formatted according to the selected candidate layout and includes information based on the transcript, the additional information associated with the online communication session, or both; causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices; and synchronizing the changes to the visualization information among the plurality of client devices during the online communication session.
An example method implemented in a data processing system includes detecting an occurrence of a trigger condition during an online communication session among a plurality of client devices associated with participants of the online communication session, the occurrence of the trigger condition indicating that a visualization of content associated with the online communication session should be generated and presented on the client devices associated with the participants of the online communication session; analyzing information associated with the online communication session to determine a meeting category associated with the online communication session, wherein analyzing the information associated with the online communication session includes generating a transcript of an audio portion of the online communication session using a transcription model, and analyzing the transcript and additional information associated with the online communication session as an input to a categorization model that outputs the meeting category associated with the online communication session; selecting a set of candidate visualization layouts from a plurality of visualization layouts in a visualization layout datastore based on the meeting category associated with the online communication session; causing a client device of a participant to the online communication session to present the set of candidate visualization layouts on a user interface of the client device; receiving an indication from the client device indicating a selected candidate layout from among the plurality of candidate visualization layouts; providing the selected candidate layout, the transcript, and the additional information associated with the online communication session as an input to a language model; obtaining visualization information as an output from the language model, wherein the visualization information is formatted according to the selected candidate layout and includes information based on the transcript, the additional information associated with the online communication session, or both; causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices, wherein the user interface provides controls that enable users of the plurality of client devices to make changes to the visualization information; and synchronizing the changes to the visualization information among the plurality of client devices during the online communication session.
An example data processing system according to the disclosure includes a processor and a memory storing executable instructions. The instructions when executed cause the processor alone or in combination with other processors to perform operations including establishing an online communication session with a first application on a plurality of client devices of participants to the online communication session; receiving an indication to generate a visualization of content of the online communication session; constructing a prompt for a language model based on a transcript of the online communication session; providing the prompt as an input to a language model; obtaining visualization information as an output from the language model, wherein the visualization information is formatted according to a layout generated by the language model and includes information based on the transcript; causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices, wherein the user interface provides controls that enable users of the plurality of client devices to make changes to the visualization information; and synchronizing the changes to the visualization information among the plurality of client devices during the online communication session.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
The drawing figures depict one or more implementations in accord with the present teachings, by way of example only, not by way of limitation. In the figures, like reference numerals refer to the same or similar elements. Furthermore, it should be understood that the drawings are not necessarily to scale.
Systems and methods for automatically generating visualizations for online communication sessions are described herein. These techniques provide a technical solution to the technical problem of how to increase participant engagement retention of information and shared understanding in online meetings conducted on an online communications platform. Many online communications platforms allow participants to an online meeting to present static or interactive content during the meeting. The inclusion of interactive content in particular can increase participant engagement. However, many meeting participants are often unaware of the ability to add such interactive content to an online meeting, unaware how to add such interactive content to the online meeting, or are aware that interactive content exists, but find it onerous to set up. The techniques herein provide a technical solution to these problems by automatically generating an interactive visualization of content associated with an online communication session and presenting the visualization to participants during the online communication session. The participants can collaborate with the other meeting participants to add to the interactive visualizations of the content. The interactive visualization is implemented as an interactive virtual whiteboard on which the participants can type, draw, write, or otherwise interact with the virtual whiteboard using a keyboard, mouse, stylus pen, touchscreen, voice, and/or other input. Initially, contents of the virtual whiteboard are automatically created by analyzing a transcript of the online communication session, chat messages associated with the online communication session, and/or other data associated with the online communication session to determine a meeting category for the online communication session and to select visualization layout templates based on the meeting category. The visualization layout templates are presented to a participant to the online communication session to select a preferred layout from among the visualization layout templates, and the contents of the visualization are automatically generated by a generative language model from the meeting transcript, the chat messages, and/or other content associated with the online communication session using a generative language model. In other implementations, the generative language model does not rely on predetermined visualization layout templates and instead creates the layout for the visualization. The contents of the visualization can also be updated automatically based on the meeting transcript, the chat messages, and/or other content associated with the online communication session during the online communication session. A technical benefit of this approach is that participant engagement and information retention can be increased. Furthermore, the automatic creation and updating of the visualization content using the generative language model can help reduce the overall computing resources associated with creating the collaborative content by generating content that is predicted to be relevant to the participants of the online communication session based on transcription, chat messages, and other content associated with the online communication session. This approach provides the participants with a starting point for further collaboration that encourages discussion and collaboration rather than starting from a blank document in which the participants struggle to develop a visualization that meets their needs, thereby consuming additional computing resources. These and other technical benefits of the techniques disclosed herein will be evident from the discussion of the example implementations that follow.
The application services platform 110 includes a request processing unit 122, artificial intelligence (AI) services 124, visualization support unit 180, moderation services 168, a web application 190, and meeting information datastore 196. The request processing unit 122 is configured to receive requests from the native application 114 of the client device 105 and/or the web application 190 of the application services platform 110. The requests may include but are not limited to requests to establish an online communication session and/or requests to create, view, and/or modify various types of electronic content, including creating visualizations of content of an online communication session as discussed in the examples which follow. In the example implementation shown in
The meeting information datastore 196 is a persistent datastore in the memory of the application services platform 110 that stores information associated with an online communication session. The meeting information datastore 196 stores meeting invite information, participant information, recordings of meeting audio and/or video, transcripts of the meeting, contents of the meeting chat, file attachment information for files shared by participants of the online communication session, visualization information for visualizations created during the online communication session, and/or other information associated with an online communication session. The visualization information can be used to populate a virtual whiteboard as discussed in the examples which follow. The meeting information datastore 196 is accessible to the native application 114, the web application 190, and/or the communications platform web application 192 to permit the participants to access the content associated with online communication session both during and after the online communication session.
The AI services 124 include transcription model 132, a meeting categorization model 130, a prompt construction unit 126, a language model 128, and a topic model 134. The transcription model 132 is trained to receive audio content from an online communication session, to identify the speech included therein, and to output a written transcript of this speech. Some implementations of the transcription model 132 can identify which participant to the online communication session is speaking and add an identifier for that participant with the speech in the transcript so that someone reading the transcript can identify who said what. The meeting categorization model 130 is a classification model that is trained to predict a meeting category from a set of meeting categories supported by the application services platform 110 for creating visualizations of the meeting content. The meeting categorization model 130 receives various types of textual input, such as but not limited to all or a portion of the meeting transcript, all or portion of the meeting chat, a number of meeting, attendee information (such as the job title, role, or other information), and/or information obtained from the calendar invite associated with the meeting. The information from the calendar invite may include but is not limited to a meeting title, a meeting description, and attached information and/or textual content extracted from the attachments to the calendar invite. The meeting categorization model 130 is trained to predict a meeting category from among the supported meeting categories based on the information available to the model. The prediction output by the meeting categorization model 130 is associated with a confidence score in some implementations, and the AI services 124 are configured to output an indication that the meeting category could not be determined if the confidence score is below a predetermined confidence score threshold. In such instances, the application services platform 110 suggests a default set of visualization layouts to the user based on the most frequently selected visualization layouts.
The language model 128 is a machine learning model trained to generate textual content in response to natural language prompts input by a user via the native application 114 or via the browser application 112. The language model 128 is implemented using a large language model (LLM) in some implementations. Examples of such models include but are not limited to a Generative Pre-trained Transformer 3 (GPT-3), or GPT-4 model. Other implementations may utilize other models or other generative models to generate textual content in response to user prompts. The language model 128 is used to generate content to be included in visualizations to be presented to users in an online communication session as discussed in the examples which follow.
The prompt construction unit 126 constructs a prompt based on the natural language prompts provided by users of the native application 114 or the web application 190 and/or additional information. The additional information may include specific information that can be used by the language model 128 for generating a visualization based on the content of the online communications session. The additional information can include a selected candidate visualization layout that indicates how the visualization should be formatted. The additional information can also include at least a portion of the transcript, at least a portion of the chat messages, and/or at least a portion of the text of any attachments shared by participants of the online communication session.
The topic model 134 is a language model that is trained to receive a textual input and to determine a topic or topics associated with the textual input. The topic model 134 is provided an excerpt of the transcript and/or the chat messages associated with an online communication session to obtain a current topic for the
The visualization support unit 180 provides visualization layout templates that can be used to create visualizations of an online communication session. The visualization support unit 180 selects candidate visualization layouts based on the predicted meeting categories output by the meeting categorization model 130. In some implementations, the generative language model 128 generates a layout for the visualizations without relying on a visualization layout template. The visualization support unit 180 also maintains trigger information for trigger conditions that cause the application services platform 110 to automatically create a visualization for an online communication session and/or to prompt a participant of the online communication session whether such a visualization should automatically be created for the online communication session. The trigger conditions are associated with a predicted intent of the participants of the online communication session under various circumstances. Additional details of the visualization support unit are discussed below with respect to
The moderation services 168 analyze natural language prompts input by the user for interacting with the language model 128 of the AI services 124, and textual content generated by the language model 128 and the transcription model 132 to ensure that potentially objectionable or offensive content is not generated or utilized by the application services platform 110. If potentially objectionable or offensive content is detected, the moderation services 168 provides a blocked content notification to the client device 105 indicating that the natural language prompt and/or the content generated by the language model 128 or the transcription model 132 included content that is blocked.
The moderation services 168 performs several types of checks on the natural language prompts entered by the user in the native application 114 or the web application 190 and/or content generated by the language model 128 or the transcription model 132. The content moderation unit 170 is implemented by a machine learning model trained to analyze the textual content of these various inputs to perform a semantic analysis on the textual content to predict whether the content includes potentially objectionable or offensive content. The language check unit 172 performs another check on the textual content using a second model configured to analyze the words and/or phrase used in textual content to identify potentially offensive language. The guard list check unit 174 is configured to compare the language used in the textual content with a list of prohibited terms including known offensive words and/or phrases. The dynamic list check unit 176 provides a dynamic list that can be quickly updated by administrators to add additional prohibited words and/or phrases. The dynamic list may be updated to address problems such as words or phrases becoming offensive that were not previously deemed to be offensive. The words and/or phrases added to the dynamic list may be periodically migrated to the guard list as the guard list is updated. The specific checks performed by the moderation services 168 may vary from implementation to implementation. If one or more of these checks determines that the textual content includes offensive content, the moderation services 168 can notify the application services platform 110 that some action should be taken.
In some implementations, the moderation services 168 generates a blocked content notification, which is provided to the client device 105. The native application 114 or the web application 190 receives the notification and presents a message on a user interface of the application that submitted the natural language prompt which could not be processed. The user interface provides information indicating why the blocked content notification was issued in some implementations. The user may attempt to refine the natural language prompt to remove the potentially offensive content. A technical benefit of this approach is that the moderation services 168 provides safeguards against both user-created and model-created content to ensure that prohibited offensive or potentially offensive content is not presented to the user in the native application 114 or the web application 190. If the moderation services 168 detects potentially objectionable or offensive content in the transcript generated by the transcription model 132, the potentially objectionable or offensive content is redacted from the transcript, but the remainder of the transcript is made available to the participants to the online communication session so that the entire transcript of the session is not lost.
The client device 105 is a computing device that may be implemented as a portable electronic device, such as a mobile phone, a tablet computer, a laptop computer, a portable digital assistant device, a portable game console, and/or other such devices in some implementations. The client device 105 may also be implemented in computing devices having other form factors, such as a desktop computer, vehicle onboard computing system, a kiosk, a point-of-sale system, a video game console, and/or other types of computing devices in other implementations. While the example implementation illustrated in
The client device 105 includes a native application 114 and a browser application 112. The native application 114 is a web-enabled native application, in some implementations, that enables users to participate in online communication sessions and collaborate with meeting participants in creating and/or modifying visualizations of content associated with the meeting. Such visualizations may be created by the application services platform 110 according to the techniques described herein. The web application 190 also enables users to participate in online communication sessions and collaborate with meeting participants in creating and/or modifying visualizations of content associated with the meeting. The browser application 112 can be used for accessing and viewing web-based content provided by the application services platform 110. In such implementations, the application services platform 110 implements one or more web applications, such as the web application 190, that enables users to participate in online communication sessions and collaborate with meeting participants in creating and/or modifying visualizations of content associated with the meeting. The application services platform 110 supports both the native application 114 and a web application 190 in some implementations, and the users may choose which approach best suits their needs.
The native application 114 in the example implementation
The remaining elements of the application services platform 110 shown in
The visualization trigger datastore 204 is another persistent datastore in the memory of the application services platform 110. The visualization trigger datastore 204 stores trigger information for trigger conditions. These trigger conditions are associated with various conditions are indicative of the intent of the participants of the application services platform 110 to have the application services platform 110 automatically generate a visualization for the online communication session. The application services platform 110 creates the visualization for an online communication session and/or prompts a participant of the online communication session whether such a visualization should be created in response to the occurrence of the trigger condition. An administrator may configure the specific trigger conditions included in the visualization trigger datastore 204 for a particular implementation. Furthermore, the visualization trigger datastore 204 may be populated with a set of default trigger conditions and the administrator may modify the default trigger conditions to suit the particular implementation. Each trigger condition identifies one or more conditions that must occur to satisfy the trigger condition and one or more actions to be taken by the application services platform 110 in response to the trigger condition being satisfied. The trigger conditions may be grouped into three categories: (1) manual start trigger conditions, (2) reactive start trigger conditions, and (3) proactive start trigger conditions. Manual start trigger conditions are based on a participant of an online communication session taking some action to cause the application services platform 110 to automatically generate a visualization for the online communication session. In a non-limiting example, the participant clicks on or otherwise activates a control in the user interface of the communications platform application implemented by the native application 114, the web application 190, or the communications platform web application 192. The activation of the control indicates to the application services platform 110 that a visualization of the content of the online communication session should be created. The reactive start trigger conditions are based on the content of the natural language prompts that the online communication session participant inputs during the online communication session. Examples of such inputs are shown in
The trigger monitoring unit 206 monitors participant inputs, the meeting transcript, the meeting chat, and other meeting related information discussed above. The trigger monitoring unit 206 accesses the visualization trigger datastore 204 to obtain the trigger condition information and analyzes the various inputs to determine whether any of the trigger conditions have been satisfied. If a trigger condition has been satisfied, the trigger monitoring unit 206 performs the one or more actions associated with that trigger condition. These actions may include initializing a new interactive visualization and/or updating the visualization information of an existing interactive visualization. Additional details of how the one or more actions are performed are described in greater detail in the examples which follow.
The candidate layout selection unit 208 receives the predicted meeting category output by the meeting categorization model 130 and identifies the candidate visualization templates from the visualization layout templates datastore 202. The candidate layout selection unit 208 ranks the candidate visualization layouts selected for the meeting category based on a frequency with which the candidate visualization layouts are selected for that meeting category and selects a set of highest-ranking candidate visualization layouts to include in the set of candidate visualization layouts. The candidate layout selection unit 208 selects a predetermined number of candidate visualization layouts to present to a participant or participants of the online communication session. In some implementations, an administrator is able to customize the predetermined number of candidate visualization layouts presented. The candidate layout selection unit 208 causes the candidate visualization layouts to be presented on a user interface of the communication platform application implemented by the native application 114, the web application 190, or the communications platform web application 192 depending upon the implementation of the application services platform 110. The participant may then select the layout of the visualization to be generated. The language model 128 of the AI services 124 is used to generate, at least in part, the content for the visualization based on the visualization template selected by the participant. Alternatively, in some implementations, the candidate layout selection unit 208 automatically selects the visualization template to be utilized for the visualization. Additional details of the generation of the visualization are described in greater detail in the examples which follow.
The visualization state unit 210 maintains the state of the visualization information so that the state of visualization information used to populate the interactive visualization can be automatically updated based on the ongoing conversation in the online communication session. The visualization state unit 210 stores information about the current topic of conversation, as determined by the topic model 134. The visualization state unit 210 obtains an except of the transcript and/or the chat messages associated with the online communication session and provides the excerpts to the topic model 134 for analysis. The visualization state unit 210 periodically checks for a change in topic in some implementations. In a non-limiting example, every m minutes, the visualization state unit 210 obtains an excerpt of the transcript and/or the chat messages reflecting the last m minutes and provides the excerpted information to the AI services 124 for analysis by the topic model 134. The current topic output by the topic model 134 is compared with the previous topic. If the topic has not changed, the visualization state unit 210 causes the trigger monitoring unit 206 to check whether a layout synchronization trigger condition has been satisfied so that the trigger monitoring unit 206 will update the visualization content of the current visualization with additional information included in the transcript and/or chat since the visualization information was last updated. Otherwise, if the topic has changed, the visualization state unit 210 provides an indication to the trigger monitoring unit 206 that a layout change trigger condition has occurred, which indicates that the topic has changed, and a new visualization can be created for the new topic.
The process 300 then continues with the trigger monitoring unit 206 monitoring the online communication session as discussed in the preceding examples to determine whether a trigger condition has been satisfied. In operation 304, the trigger monitoring unit 206 detects a manual start trigger condition has been satisfied. In operation 306, the trigger monitoring unit 206 detects that a reactive start trigger condition has been satisfied. In operation 308, the trigger monitoring unit 206 detects that a proactive start trigger condition has been satisfied. The process 300 continues to operation 312 in response to the trigger monitoring unit 206 detecting that a manual start trigger condition, or a reactive start trigger condition has been satisfied. However, the process 300 continues to operation 310 in response to the trigger monitoring unit 206 detecting that a proactive start trigger condition has been satisfied. In operation 310, a participant of the online communication session is prompted whether a visualization should automatically be generated and presented to the participants of the online communication session. The trigger monitoring unit 206 causes a prompt to be presented on a user interface of the client device 105 of the participant. The participant can then respond via the prompt with an indication that the visualization should be created or an indication that the visualization should not be created. If the participant indicates that the visualization should be created, the process 300 proceed to operation 312. Otherwise, if the participant indicates that the visualization should not be created, the trigger monitoring unit 206 continues to monitor for the occurrence of a trigger condition. The participant may be the organizer of the online communication session, or another participant selected to make the decision whether the visualization should be automatically generated. The participant who is prompted by the trigger monitoring unit 206 may be a participant in the online communication session that is currently speaking at the time that the trigger monitoring unit 206 determines that the proactive start trigger has been satisfied.
In operation 312, the candidate layout selection unit 208 obtains the candidate visualization layouts from the visualization layout templates datastore 202 and causes the client device 105 of the participant to present the candidate visualization layouts to the participant in operation 314. The participant selects a candidate visualization layout from among the candidate visualization layouts presented to the participant and the client device 105 of the participant provides an indication of the selected candidate visualization layout to the candidate layout selection unit 208.
In operation 316, the application services platform 110 generates the selected visualization. The prompt construction unit 126 generates the prompt that is provide to the language model 128 to obtain the visualization generated by the language model 128. The request processing unit 122 includes the selected candidate visualization layout as one of the inputs to the prompt construction unit 126. The prompt construction unit 126 uses the candidate visualization layout to provide context to the model as to what the visualization generated by the model should look like. The request processing unit 122 also includes at least a portion of the transcript, at least a portion of the chat messages, and/or at least a portion of the text of any attachments shared by participants of the online communication session. The prompt construction unit 126 may format the input information into a standardized format that is recognized by the language model 128. The request processing unit 122 may also provide a natural language prompt input by a participant of the online communication session to the prompt construction unit 126, and the prompt construction unit 126 includes the natural language prompt to the language model 128 to provide context to what the language model 128 regarding the visualization to be created.
In operation 318, the generated visualization is presented on a user interface of the client devices 105 of the participants of the online communication session. Depending upon the implementation, the native application 114, the web application 190, or the visualization web application 194 presents the visualization. In implementations in which the visualization web application 194 presents the generated visualization, the presentation may be within a window of the user interface of the communications platform web application 192 or of the native application 114. The participants to the online communication session can then begin interacting with and contributing to the visualization. The request processing unit 122 of the application services platform 110 can coordinate receiving the updates made by each of the participants, updating a state of the visualization information representing the visualization being shared, and pushing these changes out to the client devices 105 of the participants of the online communication session. The request processing unit 122 can also store content associated with the online communication session, including the visualization information in the meeting information datastore 196.
The trigger monitoring unit 206 can listen for layout change triggers and layout synchronization triggers once the visualization has been generated. In operation 320, the trigger monitoring unit 206 detects a layout change trigger condition has been satisfied. The layout change triggers are associated with trigger conditions associated with a change in topic in the online communication session. When the topic of conversation in the online communication session changes, the current visualization can be saved to the meeting information datastore 196, and a new visualization instantiated for the new topic of conversation. In some implementations, the participant that initiated the current visualization clicks on or otherwise activates a control on the user interface of the native application 114, the web application 190, or the communications platform web application 192. The AI services 124 includes a language model trained to analyze textual inputs, such as but not limited to the transcript and chat messages to predict a current subject matter of the online communication session. The trigger monitoring unit 206 periodically checks the current subject matter of the online communication session to determine whether a change in subject matter has occurred. In response to the determining that a subject matter change has occurred, the process 300 can return to operation 312 in which a new set of visualization suggestions are generated for the online communication session.
The trigger monitoring unit 206 detects a layout synchronization trigger condition has been satisfied. The layout synchronization triggers cause the trigger monitoring unit 206 to synchronize the contents of the visualization with the current state of the online meeting session. Updates to the transcript, the meeting chat, and any changes that were made manually to the visualization by participants of the online communication session are considered when updating the visualization. The layout synchronization is intended to include any new ideas that were discussed in the transcript and/or the meeting chat in an updated version of the visualization. The manual edits made to the visualization to add, edit, or delete ideas by participants are also considered so that ideas that have already been added by the participants aren't duplicated and ideas deleted by the participants are not reintroduced in the new version of the visualization. The layout synchronization triggers may be automatic, semi-automatic, or manual. The automatic trigger condition is satisfied if any changes to the transcript and/or meeting chat are detected. The contents of the visualization are updated automatically and substantially in real time. For the semi-automatic synchronization trigger condition, the trigger monitoring unit 206 checks periodically whether the participants have made any changes to the contents of the visualization. For example, the trigger monitoring unit 206 may check every few minutes whether the transcription and/or the chat has changed and update the visualization accordingly. The current meeting transcript, meeting chat, and manual edits made to the visualization by the participants of the online communication application are provided to the prompt construction unit 126 of the AI services 124 to construct a prompt to cause the language model 128 to generate an updated version of the visualization information to be used to populate a virtual whiteboard to present to the participants of the online communication session. The prompt construction unit 126 formats the transcript information, the chat information, and the manual edits information into a standardized format that is recognized by the language model 128. The prompt construction unit 126 may also include the previously generated visualization information output by the language model 128 to provide the model with the previous state of the visualization before the manual edits were made by the participants. Any new ideas introduces in the meeting transcript and/or the meeting chat are included in the updated version of the visualization information output by the language model 128. The updated visualization information is then presented to the participants of the online communication session via the native application 114, the web application 190, and/or the communications platform web application 192.
In an alternate implementation of the process 300 shown in
In another alternate implementation of the process 300 shown in
The meeting category may also be provided as one of the inputs to the language model 128 in the implementations discussed above in which the language model 128 generates the layout for the visualization. The meeting category can be included as part of the prompt provided as an input to the language model 128. This approach provides the language model 128 with additional context that may help improve the layout generated by the language model 128 for the online communication session.
The process 500 includes an operation 502 of detecting an occurrence of a trigger condition during an online communication session among a plurality of client devices associated with participants of the online communication session. The occurrence of the trigger condition indicates that a visualization of content associated with the online communication session should be generated and presented on the client devices 105 associated with the participants of the online communication session. As discussed in the preceding examples, the trigger monitoring unit 206 monitors various aspects of the online communication session to determine whether a trigger condition is satisfied that indicates that a visualization should be created for the online communication session.
The process 500 includes an operation 504 of analyzing information associated with the online communication session to determine a meeting category associated with the online communication session. Analyzing the information associated with the online communication session includes generating a transcript of an audio portion of the online communication session using a transcription model, and analyzing the transcript and additional information associated with the online communication session as an input to a categorization model that outputs the meeting category associated with the online communication session. The transcription model 132 generates a transcript of the online communication session, and the categorization model 130 analyzes the transcript and additional information to determine a meeting category for the online communication session. The additional information can include all or portion of the meeting chat, a number of meeting, attendee information (such as the job title, role, or other information), and/or information obtained from the calendar invite associated with the meeting. The information from the calendar invite may include but is not limited to a meeting title, a meeting description, and attached information and/or textual content extracted from the attachments to the calendar invite. The meeting categorization model 130 is trained to predict a meeting category from among the supported meeting categories based on the information available to the model.
The process 500 includes an operation 506 of selecting a set of candidate visualization layouts from a plurality of visualization layouts in a visualization layout datastore based on the meeting category associated with the online communication session and an operation 508 of causing a client device of a participant to the online communication session to present the set of candidate visualization layouts on a user interface of the client device 105. The candidate layout selection unit 208 selects the set of candidate visualization layouts from the visualization layout templates datastore 202, and the set of candidate visualization layouts are presented on a user interface of the client device 105 of the participant of the online communication session.
The process 500 includes an operation 510 of receiving an indication from the client device indicating a selected candidate layout from among the plurality of candidate visualization layouts. As discussed in the preceding examples, the participant presented with the set of candidate visualization layouts can select one of these candidates and an indication of this selection is provided to the application services platform 110 by the client device 105 of the participant.
The process 500 includes an operation 512 of providing the selected candidate layout, the transcript, and the additional information associated with the online communication session as an input to a language model 128, and an operation 514 of obtaining visualization information as an output from the language model 128. The visualization information is formatted according to the selected candidate layout and includes information based on the transcript, the additional information associated with the online communication session, or both. The visualization information can be used to populate a virtual whiteboard. The prompt construction unit 126 creates the prompt for the language model 128 based on the selected candidate layout, the transcript, and the additional information associated with the online communication session as discussed in the preceding examples.
The process 500 includes an operation 516 of causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices 105. The interactive workspace is a virtual whiteboard in some implementations, and the visualization information is used to populate the virtual whiteboard. The user interface provides controls that enable users of the plurality of client devices to make changes to the visualization information. As discussed in the preceding examples, the participants to the online communication session are able to collaborate with other meeting participants in creating and/or modifying visualizations of content associated with the meeting.
The process 500 includes an operation 518 of synchronizing the changes to the visualization information among the plurality of client devices during the online communication session. As discussed in the preceding examples, the trigger monitoring unit 206 of the application services platform 110 detects changes are made to the visualization information and the request processing unit 122 pushes these changes out to the client devices 105 of the participants via native application 114, the web application 190, or the communications platform web application 192.
The process 600 includes an operation 602 of establishing an online communication session with a first application on a plurality of client devices of participants to the online communication session. The application services platform 110 facilitates establishing the online communication session with the plurality of client device 105 of the participants of the online communication session.
The process 600 includes an operation 604 of receiving an indication to generate a visualization of content of the online communication session. The occurrence of the trigger condition indicates that a visualization of content associated with the online communication session should be generated and presented on the client devices 105 associated with the participants of the online communication session. As discussed in the preceding examples, the trigger monitoring unit 206 monitors various aspects of the online communication session to determine whether a trigger condition is satisfied that indicates that a visualization should be created for the online communication session.
The process 600 includes an operation 606 of constructing a prompt for a language model 128 based on a transcript of the online communication session and an operation 608 of providing the prompt to the language model 128 as an input. The prompt construction unit 126 includes at least a portion of the transcript in the prompt in some implementations. The prompt may also include additional information, such as a meeting category for the online communication session, chat messages for the online communication session, and/or additional information. The categorization model 130 analyzes the transcript and additional information to determine a meeting category for the online communication session. Alternatively, in some implementations, the prompt includes an instruction of the constructing the prompt to include instructions to the language model to generate a set of candidate visualization layouts based on the transcript in which each of the candidate visualization layouts is formatted according to a different layout generated by the language model and include information based on the transcript.
The process 600 includes an operation 610 of obtaining visualization information as an output from the language model. The visualization information is formatted according to a layout generated by the language model and includes information based on the transcript.
The process 600 includes an operation 612 of causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices, wherein the user interface provides controls that enable users of the plurality of client devices to make changes to the visualization information. The interactive workspace is a virtual whiteboard in some implementations, and the visualization information is used to populate the virtual whiteboard. The user interface provides controls that enable users of the plurality of client devices to make changes to the visualization information. As discussed in the preceding examples, the participants to the online communication session are able to collaborate with other meeting participants in creating and/or modifying visualizations of content associated with the meeting.
The process 600 includes an operation 614 of synchronizing the changes to the visualization information among the plurality of client devices during the online communication session. As discussed in the preceding examples, the trigger monitoring unit 180 of the application services platform 110 detects changes are made the visualization information and the request processing unit 122 pushes these changes out to the client devices 105 of the participants via native application 114, the web application 190, or the communications platform web application 192.
The process 700 includes an operation 702 of detecting an occurrence of a trigger condition during an online communication session among a plurality of client devices associated with participants of the online communication session. The occurrence of the trigger condition indicates that a visualization of content associated with the online communication session should be generated and presented on the client devices 105 associated with the participants of the online communication session. As discussed in the preceding examples, the trigger monitoring unit 206 monitors various aspects of the online communication session to determine whether a trigger condition is satisfied that indicates that a visualization should be created for the online communication session.
The process 700 includes an operation 704 selecting a candidate visualization layout from a plurality of visualization layouts in a visualization layout datastore based on the meeting category associated with the online communication session. The candidate layout selection unit 208 selects the candidate visualization layout from the visualization layout templates datastore 202.
The process 700 includes an operation 706 constructing a prompt for a language model based on the selected candidate visualization layout and a transcript of the online communication session and an operation 708 providing the prompt as an input to a language model 128. The prompt construction unit 128 constructs the prompt for the language model 128 as discussed in the preceding examples and provides the prompt to the language model 128.
The process 700 includes an operation 710 obtaining visualization information as an output from the language model and an operation 712 causing the plurality of client devices associated with the participants of the online communication session to present the visualization information in an interactive workspace on a user interface of the plurality of client devices. The interactive workspace is a virtual whiteboard in some implementations, and the visualization information is used to populate the virtual whiteboard. The visualization information is formatted according to the selected candidate layout and includes information based on the transcript, and the additional information associated with the online communication session, or both. The visualization information is formatted according to the selected candidate layout and includes information based on the transcript, the additional information associated with the online communication session, or both.
The process 700 includes an operation 714 synchronizing the changes to the visualization information among the plurality of client devices during the online communication session. As discussed in the preceding examples, the trigger monitoring unit 206 of the application services platform 110 detects changes are made to the visualization information and the request processing unit 122 pushes these changes out to the client devices 105 of the participants via native application 114, the web application 190, or the communications platform web application 192.
The detailed examples of systems, devices, and techniques described in connection with
In some examples, a hardware module may be implemented mechanically, electronically, or with any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is configured to perform certain operations. For example, a hardware module may include a special-purpose processor, such as a field-programmable gate array (FPGA) or an Application Specific Integrated Circuit (ASIC). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations and may include a portion of machine-readable medium data and/or instructions for such configuration. For example, a hardware module may include software encompassed within a programmable processor configured to execute a set of software instructions. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (for example, configured by software) may be driven by cost, time, support, and engineering considerations.
Accordingly, the phrase “hardware module” should be understood to encompass a tangible entity capable of performing certain operations and may be configured or arranged in a certain physical manner, be that an entity that is physically constructed, permanently configured (for example, hardwired), and/or temporarily configured (for example, programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering examples in which hardware modules are temporarily configured (for example, programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where a hardware module includes a programmable processor configured by software to become a special-purpose processor, the programmable processor may be configured as respectively different special-purpose processors (for example, including different hardware modules) at different times. Software may accordingly configure a processor or processors, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time. A hardware module implemented using one or more processors may be referred to as being “processor implemented” or “computer implemented.”
Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (for example, over appropriate circuits and buses) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory devices to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output in a memory device, and another hardware module may then access the memory device to retrieve and process the stored output.
In some examples, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by, and/or among, multiple computers (as examples of machines including processors), with these operations being accessible via a network (for example, the Internet) and/or via one or more software interfaces (for example, an application program interface (API)). The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across several machines. Processors or processor-implemented modules may be in a single geographic location (for example, within a home or office environment, or a server farm), or may be distributed across multiple geographic locations.
The example software architecture 802 may be conceptualized as layers, each providing various functionality. For example, the software architecture 802 may include layers and components such as an operating system (OS) 814, libraries 816, frameworks 818, applications 820, and a presentation layer 844. Operationally, the applications 820 and/or other components within the layers may invoke API calls 824 to other layers and receive corresponding results 826. The layers illustrated are representative in nature and other software architectures may include additional or different layers. For example, some mobile or special purpose operating systems may not provide the frameworks/middleware 818.
The OS 814 may manage hardware resources and provide common services. The OS 814 may include, for example, a kernel 828, services 830, and drivers 832. The kernel 828 may act as an abstraction layer between the hardware layer 804 and other software layers. For example, the kernel 828 may be responsible for memory management, processor management (for example, scheduling), component management, networking, security settings, and so on. The services 830 may provide other common services for the other software layers. The drivers 832 may be responsible for controlling or interfacing with the underlying hardware layer 804. For instance, the drivers 832 may include display drivers, camera drivers, memory/storage drivers, peripheral device drivers (for example, via Universal Serial Bus (USB)), network and/or wireless communication drivers, audio drivers, and so forth depending on the hardware and/or software configuration.
The libraries 816 may provide a common infrastructure that may be used by the applications 820 and/or other components and/or layers. The libraries 816 typically provide functionality for use by other software modules to perform tasks, rather than interacting directly with the OS 814. The libraries 816 may include system libraries 834 (for example, C standard library) that may provide functions such as memory allocation, string manipulation, file operations. In addition, the libraries 816 may include API libraries 836 such as media libraries (for example, supporting presentation and manipulation of image, sound, and/or video data formats), graphics libraries (for example, an OpenGL library for rendering 2D and 3D graphics on a display), database libraries (for example, SQLite or other relational database functions), and web libraries (for example, WebKit that may provide web browsing functionality). The libraries 816 may also include a wide variety of other libraries 838 to provide many functions for applications 820 and other software modules.
The frameworks 818 (also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications 820 and/or other software modules. For example, the frameworks 818 may provide various graphic user interface (GUI) functions, high-level resource management, or high-level location services. The frameworks 818 may provide a broad spectrum of other APIs for applications 820 and/or other software modules.
The applications 820 include built-in applications 840 and/or third-party applications 842. Examples of built-in applications 840 may include, but are not limited to, a contacts application, a browser application, a location application, a media application, a messaging application, and/or a game application. Third-party applications 842 may include any applications developed by an entity other than the vendor of the particular platform. The applications 820 may use functions available via OS 814, libraries 816, frameworks 818, and presentation layer 844 to create user interfaces to interact with users.
Some software architectures use virtual machines, as illustrated by a virtual machine 848. The virtual machine 848 provides an execution environment where applications/modules can execute as if they were executing on a hardware machine (such as the machine 900 of
The machine 900 may include processors 910, memory 930, and I/O components 950, which may be communicatively coupled via, for example, a bus 902. The bus 902 may include multiple buses coupling various elements of machine 900 via various bus technologies and protocols. In an example, the processors 910 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 912a to 912n that may execute the instructions 916 and process data. In some examples, one or more processors 910 may execute instructions provided or identified by one or more other processors 910. The term “processor” includes a multicore processor including cores that may execute instructions contemporaneously. Although
The memory/storage 930 may include a main memory 932, a static memory 934, or other memory, and a storage unit 936, both accessible to the processors 910 such as via the bus 902. The storage unit 936 and memory 932, 934 store instructions 916 embodying any one or more of the functions described herein. The memory/storage 930 may also store temporary, intermediate, and/or long-term data for processors 910. The instructions 916 may also reside, completely or partially, within the memory 932, 934, within the storage unit 936, within at least one of the processors 910 (for example, within a command buffer or cache memory), within memory at least one of I/O components 950, or any suitable combination thereof, during execution thereof. Accordingly, the memory 932, 934, the storage unit 936, memory in processors 910, and memory in I/O components 950 are examples of machine-readable media.
As used herein, “machine-readable medium” refers to a device able to temporarily or permanently store instructions and data that cause machine 900 to operate in a specific fashion, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, cache memory, network-accessible or cloud storage, other types of storage and/or any suitable combination thereof. The term “machine-readable medium” applies to a single medium, or combination of multiple media, used to store instructions (for example, instructions 916) for execution by a machine 900 such that the instructions, when executed by one or more processors 910 of the machine 900, cause the machine 900 to perform and one or more of the features described herein. Accordingly, a “machine-readable medium” may refer to a single storage device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” excludes signals per se.
The I/O components 950 may include a wide variety of hardware components adapted to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components 950 included in a particular machine will depend on the type and/or function of the machine. For example, mobile devices such as mobile phones may include a touch input device, whereas a headless server or IoT device may not include such a touch input device. The particular examples of I/O components illustrated in
In some examples, the I/O components 950 may include biometric components 956, motion components 958, environmental components 960, and/or position components 962, among a wide array of other physical sensor components. The biometric components 956 may include, for example, components to detect body expressions (for example, facial expressions, vocal expressions, hand or body gestures, or eye tracking), measure biosignals (for example, heart rate or brain waves), and identify a person (for example, via voice-, retina-, fingerprint-, and/or facial-based identification). The motion components 958 may include, for example, acceleration sensors (for example, an accelerometer) and rotation sensors (for example, a gyroscope). The environmental components 960 may include, for example, illumination sensors, temperature sensors, humidity sensors, pressure sensors (for example, a barometer), acoustic sensors (for example, a microphone used to detect ambient noise), proximity sensors (for example, infrared sensing of nearby objects), and/or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components 962 may include, for example, location sensors (for example, a Global Position System (GPS) receiver), altitude sensors (for example, an air pressure sensor from which altitude may be derived), and/or orientation sensors (for example, magnetometers).
The I/O components 950 may include communication components 964, implementing a wide variety of technologies operable to couple the machine 900 to network(s) 970 and/or device(s) 980 via respective communicative couplings 972 and 982. The communication components 964 may include one or more network interface components or other suitable devices to interface with the network(s) 970. The communication components 964 may include, for example, components adapted to provide wired communication, wireless communication, cellular communication, Near Field Communication (NFC), Bluetooth communication, Wi-Fi, and/or communication via other modalities. The device(s) 980 may include other machines or various peripheral devices (for example, coupled via USB).
In some examples, the communication components 964 may detect identifiers or include components adapted to detect identifiers. For example, the communication components 964 may include Radio Frequency Identification (RFID) tag readers, NFC detectors, optical sensors (for example, one- or multi-dimensional bar codes, or other optical codes), and/or acoustic detectors (for example, microphones to identify tagged audio signals). In some examples, location information may be determined based on information from the communication components 964, such as, but not limited to, geo-location via Internet Protocol (IP) address, location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and/or signal triangulation.
In the preceding detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.
While various embodiments have been described, the description is intended to be exemplary, rather than limiting, and it is understood that many more embodiments and implementations are possible that are within the scope of the embodiments. Although many possible combinations of features are shown in the accompanying figures and discussed in this detailed description, many other combinations of the disclosed features are possible. Any feature of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment unless specifically restricted. Therefore, it will be understood that any of the features shown and/or discussed in the present disclosure may be implemented together in any suitable combination. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.
While the foregoing has described what are considered to be the best mode and/or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.
Unless otherwise stated, all measurements, values, ratings, positions, magnitudes, sizes, and other specifications that are set forth in this specification, including in the claims that follow, are approximate, not exact. They are intended to have a reasonable range that is consistent with the functions to which they relate and with what is customary in the art to which they pertain.
The scope of protection is limited solely by the claims that now follow. That scope is intended and should be interpreted to be as broad as is consistent with the ordinary meaning of the language that is used in the claims when interpreted in light of this specification and the prosecution history that follows and to encompass all structural and functional equivalents. Notwithstanding, none of the claims are intended to embrace subject matter that fails to satisfy the requirement of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a way. Any unintended embracement of such subject matter is hereby disclaimed.
Except as stated immediately above, nothing that has been stated or illustrated is intended or should be interpreted to cause a dedication of any component, step, feature, object, benefit, advantage, or equivalent to the public, regardless of whether it is or is not recited in the claims.
It will be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. Relational terms such as first and second and the like may be used solely to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “a” or “an” does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, subsequent limitations referring back to “said element” or “the element” performing certain functions signifies that “said element” or “the element” alone or in combination with additional identical elements in the process, method, article, or apparatus are capable of performing all of the recited functions.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.
