Patent Application
20240127818
The following extrinsic publicly-available documents, white papers and research reports are incorporated in part, if noted specifically, or in their entireties absent partial notation, for their teachings regarding methods for visualization of conversations, turn determination in conversations, representations of spoken conversations, turn-taking modeling and theory:
For the purposes of this disclosure, these references will be referred to by their year of publication and the last name of the first listed author (or sole author).
The present invention relates to computer-based analysis and visual presentation of information regarding transcribed voice conversations.
Voice transcripts of spoken conversation are becoming more common due to the easy access to audio capture devices and the availability of accurate and efficient speech-to-text processes. Such transcripts can be captured anywhere that conversations occur between two or more people. Examples include, but are not limited to, conversations in contact centers, transcripts of meetings, archives of social conversation, and closed-captioning of television content and video interviews.
These transcripts can be utilized in a number of ways. They can be reviewed directly by people or further analyzed and indexed by automated processes, for example to label regions of meaning or emotional affect.
In some cases, the digitized audio will be retained alongside the transcript in computer memory or digital computer files. In other cases, the transcript may be retained and the audio discarded or archived separately.
A visualization is created automatically by a computer in a format resembling a vertically-scrollable text-messaging user interface by segmenting the voice transcript into phrases, resolving how to indicate visually or to suppress periods of overlapping discussion (overtalk, interruption, etc.) by applying one or more rules, transformations, or both, and outputting the visualization onto a computer display device, into a printable or viewable report, or both.
The figures presented herein, when considered in light of this description, form a complete disclosure of one or more embodiments of the invention, wherein like reference numbers in the figures represent similar or same elements or steps.
The present inventors have recognized several shortcomings in processes of the state-of-the-art technologies for producing human-readable visualizations on computer screens and other computer output types (printers, etc.) of conversations. The following paragraphs describe some of these existing systems, the shortcomings which the present inventors have recognized, and the unmet needs in the relevant arts.
Current Methods of Visualizing Voice Transcripts. For the purposes of this disclosure, a “voice transcript” or “transcript” will refer to a written description representing an audio recording, typically stored in an electronic text-based format. The transcript may have been created by a computer transcriber, by a human, or by both. Similarly, conversational voice transcripts will refer to transcripts of audio recordings of two or more speakers having a conversation between the speakers.
Voice transcripts of spoken conversation are becoming more common due to the easy access to audio capture devices and the availability of accurate and efficient speech-to-text processes. Such transcripts can be captured anywhere that conversations occur between two or more people. Examples include, but are not limited to, conversations in contact centers, transcripts of meetings, archives of social conversation, and closed-captioning of television content and video interviews.
These transcripts can be utilized in a number of ways. They can be reviewed directly by people or further analyzed and indexed by automated processes, for example to label regions of meaning or emotional affect.
In some cases, the digitized audio will be retained alongside the transcript. In other cases, the transcript may be retained and the audio discarded or archived separately.
When displaying voice conversations, many existing computer applications display a horizontal orientation of an audio waveform with one row per speaker.
As illustrated 200 in
Current methods for visualizing text conversations. With the high consumer adoption of instant messaging applications such as Facebook Messenger™ provided by Meta Platforms, Inc., of Menlo Park, California, USA, and text messaging applications such as short message service (SMS), users have become familiar with the visual representation on computer, tablet and smartphone displays which provide interactive vertically-scrolling text conversations that are input on mobile devices or computers with keyboards, as shown 400 in
The present inventors have recognized an unmet need in the art regarding these computer-based depictions of voice-based conversational transcripts in that they do not necessarily lend themselves towards the familiar vertically-scrolling messaging format due to several differences in how voice conversations unfold compared to how text messaging unfolds over time. One such difference is that, in voice conversations, it is normal for participants to speak at the same time (simultaneously), which we will refer to as overtalk. However, overtalk does not occur in text-based messaging conversations because contributions are prepared (typed) by each speaker and then instantly contributed in their entireties at a particular time in the conversation. Therefore, the digital record of a text message conversation is already encapsulated into time-stamped and chronologically-separated contributions, whereas voice-based conversations do not exhibit this inherent formatting.
Turn Taking And Overlapping Speech. In text messaging conversations, speakers indicate completion of their ‘turn’ by hitting ‘enter’ or pressing a ‘send’ button. Overtalk is avoided by the mechanism provided to enter a contribution into the conversation. In spoken conversations, however, speakers do not always wait for the other person to end what they are saying before speaking themselves. They talk over one another frequently, as discussed by Schegloff (2000). Turn-taking phenomena are varied and well studied. Using the taxonomy identified in Chowdhury (2017), key turn-taking phenomena highlighted in the literature includes:
The signals that characterize these phenomena are a complex mix of pitch, stress, pausing, and syntax. More than one such phenomena can be occurring simultaneously, and different speakers have different habitual turn-taking strategies and habits.
Most theories of turn-taking recognize the importance of the transition relevance place (TRP). These are points in time within the conversation of possible completion (or potential end) of an utterance. Smooth speaker transitions and non-competitive overlaps generally occur at or around TRPs.
Speech-to-text services. The previous section highlighted why it is a non-trivial problem to identify the start and end of speaker turns in spoken conversation. For this reason many speech-to-text processes do not even attempt this task, leaving it up to the client application to make such decisions. Instead, the speech-to-text processes recognize each speaker independently and output a sequence of words for each speaker with the start and end timings for each word, such as the example conversation shown in TABLE 1 between an agent and a client.
In the typical example of the output of a speech-to-text service shown in TABLE 1, each word is individually recognized and has the following four attributes: start time, end time, speaker, and text. The actual format may be in any structured text format such as comma separated values (CSV), JSON or YAML. It is also known to those skilled in the art that speech-to-text services may return alternate interpretations of the same conversation. For example, an acyclic graph of possible words and other tokens such as those representing silence or other paralinguistic phenomena, may be returned for each speaker with associated transition probabilities and start and end timings. It is known by those skilled in the art how to map such a representation into the tabular form presented in TABLE 1. For example, Dijkstra's process may be used to find the lowest cost path through the graph.
Aligning visualizations with perceptions of spoken dialog. The nature of turn overlaps and interruptions means that the meaning of a user utterance may be spread across multiple phrases in time.
In spoken conversation, the brain has evolved to mentally ‘edit’ spoken dialog and restructure it for maximal comprehension. We mentally join words or phrases that carry related meaning and edit-out interruptions that impart little or no extra meaning. Examples of phenomena that break-up the conversation include back-channels, back-offs, self-repairs and silent or filled pauses used for planning. In the presence of such phenomena, conversants or listeners continue to perceive the conversation evolving in an orderly fashion as long as there is not a break-down in the actual communication.
When these phenomena are reproduced in visualizations of spoken dialog, users do not have the same mental apparatus to quickly edit and interpret what they are seeing. This task becomes even harder when the speech-to-text engine also introduces recognition errors—forcing the user to also interpret missing or mentally replace substituted words. Users either need to either develop new skills or the visualization needs to present and restructure information in a way that helps to reduce the cognitive demand of the task, whereas this is not a mental process common among users of visual representations of transcripts of spoken conversations. The interpretation and understanding task becomes even more difficult when there are 3 or more (N) speakers involved in the conversation simultaneously.
Current approaches to turn taking segmentation in voice transcripts. Current approaches to the detection of turn boundaries in spoken conversations typically seek pauses between words or phrases from a single speaker. If they are above a certain time threshold then a turn boundary is considered to be present.
An example of the current state of the art would be to gather together words that have contiguous timing at word boundaries. For example, in TABLE 1 word “I” (w4) can be seen to end at exactly the same time that the word ‘was’ (w5) begins. There is, however, no guarantee that the speech-to-text process will (or even should) deliver contiguous word timings. Notice, for example, there is a 120 ms gap between the end of ‘failed’ (w2) and ‘and’ (w3). There is also a 119 ms gap between ‘see’ (w22) and ‘so’ (w23). Many of the words have much smaller gaps between them, for example, the 20 ms gap between ‘it's’ (w1) and ‘failed’ (w2). To make this approach workable it is a known practice to join together words that have less than a fixed gap between their start and end times (for example, less than 150 ms).
This approach is dependent on the availability of the start and end times for each word. These are not always present. The approach also does not take into account the information that is available from the other speaker. This approach also makes the assumption that utterances from the same speaker that are separated by a significant pause are not part of the same turn.
Summary of the Shortcomings of the Existing Technologies. As such, the foregoing paragraphs describe some, but not all, of the limitations to the existing technologies, to which the present invention's objectives are directed to solve, improve and overcome. The remaining paragraphs disclose one or more embodiments of the present invention.
A New Process for Rendering Conversation Visualizations. Referring now to
Gathering Words into Phrases. In at least one aspect of the present invention, a computer-performed process has the ability to gather words together from a transcript by a given speaker into phrases, where the transcription minimally contains the transcribed words, the timing for each word, and speaker attribution for each word, such as the process output of TABLE 1 or its equivalent.
In at least one embodiment of the present invention, the computer-performed process uses alternation between speakers as an extra signal for potential turn boundaries. In one such embodiment the computer-performed process first orders words by start time, regardless of which speaker they are from. The computer-performed process then joins together sequences of words in this ordered set where the speaker remains the same. With reference to the example transcript provided earlier in TABLE 1, the rows (or records) of this table (or database) is sorted by the computer-performed process according to the contents of the start time column ‘start_ms’. The rows in the table are designated as to which speaker, ‘agent’ or ‘client’, contributed each word by the entry in the column ‘speaker’. Contiguous runs of the same speaker are then joined together into phrases which we label P1 through P11, as shown in the column ‘phrase_Ix’.
The result is shown 600 in the swim-lane style visualization of
Note how this process forces alternation between speakers, for example phrase P1 was uttered by the client, phrase P2 was uttered by the agent, phrase P3 by the client etc., Note also that this process does not require any knowledge of the end-times of words. It therefore works with speech-to-text process output formats that only annotate words with start-times. If the output of the speech-to-text process is already ordered according to time stamps, then the new process does not even require start-times. Further, even though the present examples are given relative to two conversing parties, the new process also works with more than two speakers.
This conversant alternation is not a necessary feature for effective visualization. but it is a prerequisite for further embodiments of the invention as described herein. An extension of the at least one embodiment may be to further join together contiguous phrases for the visualization as described above. Considering
Some speech-to-text services may join words to phrases prior to outputting a result. In such cases the approach above is still relevant and can be used to join shorter phrases together into longer phrases where relevant.
Chat-Style Vertical Visualization Generator Process, The use of horizontal swim-lanes to present computer-based visualizations of spoken conversation are known to those skilled in the art. They are used in existing tools such as contact center conversational analytics platforms. In order to view a conversation, which may be very long, the user must scroll from left to right. The foregoing examples and figures merely showed simple scenarios of just two speakers and about 11 phrases. In practice, actual conversations may have many more parties and many, many more phrases. At least one objective of the present invention is to present the same transcribed conversations into a chat-style visualization, which not only is more intuitive to modern users of text-based messaging services, but also provides for infinite vertical scrolling which represents longer conversations in an easier to understand format.
According to at least one embodiment of the present invention, after the words have been automatically joined into phrases, a chat-style vertical visualization of the conversation is automatically generated, and optionally, presented to the user on a computer display, printed report, or other human-machine interface output device.
As described above the phrases P5, P7 and P9 are joined for visual presentation, as are the phrases P6 and P6. This closely mimics the style of user interfaces used to view text conversations such as SMS messages on mobile devices or contact center chat communications between customers and agents in web-browsers. In order to view these conversations the user scrolls vertically on the computer display. Vertical scrolling is much more common than horizontal scrolling in computer applications on current operating systems such as Windows from Microsoft, Android from Google, or IOS from Apple.
By combining the features of joining words into phrases, time scaling the start and end of the text boxes, the use of overlapped lanes for the user, this invention demonstrates how the visualization of spoken dialog can be adjusted to help the user mentally edit the dialog to understand its meaning.
De-emphasis of Back-channels and Back-offs. In at least one embodiment of the present invention, a further novel feature de-emphasizes or elides certain types of time-overlapping speech contributions to present a conversation that is easier to digest visually.
According to another feature available to some embodiments, the back-channel and the backed-off utterances in a conversation are completely elided from the conversation as shown 900 in
In this enhanced embodiment, the removal of overlapping speech removes the need for an accurate time-scale for the vertical axis or indeed the need to represent the presence of overlapping speech in the visualization. The automated process used to detect the presence of back-channels and back-offs is described in in the following paragraphs regarding classifying interjections.
In this embodiment the associated visualization can now be interleaved in the same manner as a text conversation. The user can then interpret this conversation in much the same way as a text conversation and no longer needs to learn any new skills to perceive the content.
Re-ordering of Overlapping Speech. The alternation process described above is helpful for the detection of turn-boundaries at transition relevant points but there are situations where customers both speak over one another in a sustained manner. TABLE 2 shows an example of the output of the speech-to-text engine for a fragment of dialog where this occurs.
In still other embodiments, the process may further reorder and gather the utterances. The process takes as its input a digital table of utterances or phrases from one conversation, such as the example shown in TABLE 1. Following the same process used to create the data visualized in
With this set of alternating utterances, the process selects a start-turn and considers this turn and the three subsequent alternating turns. It decides whether to apply a transformation to this set of utterances or not. For example it might join utterances in the set. It then returns the location of the next unaltered turn as a start point and the automated process repeats the comparison. In this way it works through the whole dialog until there are no utterances left.
This process is repeated for a few iterations starting on a different turn each time to ensure that odd and even turns are treated equally and no possible merges are overlooked. This iteration process is described in the following example pseudo-code:
In this pseudocode example, the function concat_utterances considers the four turns in utt_table starting at turn. The process then optionally transforms these utterances and returns a turn index which is moved forwards in the dialog.
The outer loop of this example pseudocode continues to call concat_utterances until there are less than four turns left in the dialog being processed. Then, the next iteration starts at the start of the dialog again with the start-turn for this iteration. In one particular embodiment, four passes are performed starting at turn 0, 1, 2, and 1 again. Other embodiments may be configured to perform more or fewer passes. Other start turns and numbers of iterations are possible and the automated process is relatively insensitive to the choice of the start-turns. Multiple iterations are important to make sure that all possible opportunities to gather turns together are discovered.
From the example in TABLE 2, each of the four utterances that are input to concat_utterances for comparison have the following three values:
Note that the times in the example data in TABLE 2 are expressed in milliseconds since the start of the interaction but they could be any representation of time. Additional parameters are derived by the process from these measures, such as those shown in
The function NumWords counts the number of words delimited by space between words in the text. Other tokenization criteria could be used in other embodiments. An ordered sequence of potential transformation rules are then considered. Each rule has a set of trigger conditions and an associated transform.
TABLE 3 shows a set of rules used in at least one embodiment, which are executed in order from top to bottom in this example.
For a transform to be executed, all of its triggers must be true. When a transformation rule is found to be true then the four utterances are transformed into a pair of utterances according to the transform and the turn counter is incremented by four. This means that these four utterances will not be considered again in this pass.
If no rules are triggered, then no transformation is made and the turn counter is incremented by two. This means that the last two utterances of this set of four (A2 and B2) will become the first two utterances (A1 and B1) for the next comparison step in this iteration.
The example rules of TABLE 3 refer to a set of example parameters which are described in TABLE 4 with the example transform patterns described in TABLE 5. The rules show two at least transform patterns—embed and join.
The embed transform pattern concatenates one pair of utterances from one of the speakers into a longer utterance but also embeds one of the utterances from the other speaker into it as an ‘interjection’. The other utterance is left unchanged. Either B1 is injected into A1+A2 or A2 is injected into B1+B2. In the pattern where B1 is considered to be an interjection into A1 and A2, the utterance B1 can be thought of as either a back-channel or a back-off which overlaps the combined turn of A1 and A2 but does not break its meaning. Thus we treat A1 and A2 as a single utterance and note the interjection of B1 but treat A1 and A2 as a single combined utterance.
The join transform pattern concatenates the two pairs of utterances from each speaker into two longer utterances, one from each speaker. This can be thought of as joining A2 to the end of A1 and joining B1 to the start of B2. Words or phrases that were broken into two utterances are now joined as a single utterance. The new bigger utterances keep the start and end times of the two utterances that were joined. The timing of the gap between the two utterances that were joined is lost. The two new bigger utterances from the two speakers can still overlap in time and alternation is preserved.
TABLE 6 and
It can be seen that the resulting visualized and displayed dialog is easier for a user to read and understand quickly because much of the complexity found in the original transcribed text data is removed and converted into graphic relationships, however, the visualization still retains the structure of the dialog and the intent of the speakers.
Missing End-Times. As has be noted in previous paragraphs, some speech-to-text processes only return (output) the start-time of an utterance and not the end-time. In the absence of end times being provided in the transcription data received by an embodiment of the present invention, the new process uses the start time of the next utterance in the input data table as an approximation for the end-time of the current utterance., e.g. a1_end=b2_start and b1_end=a2_start. This approximation assumes that the utterances are truly alternating with no overlap and no pause between them. The duration parameters become the delta (difference) between the start times of the alternating utterances and the hole parameters are the same as the duration parameters.
TABLE 7 shows how the rules in TABLE 4 are modified when the end times are subject to this approximation. In this example, the rules same_span and other_span are not modified because they do not depend on the end times. The rules b1_interject, a2_interject, end_join, begin_join do use the start and end times and are automatically disabled if the value for floor_yield is non-zero. This is a very useful embodiment according to the present invention under these circumstances. The process no longer attempts to detect interjections or beginning or ending overlaps in such a situation and embodiment.
Classifying interjections. The transformation rules b1_interject and a2_interject detect short utterances from one speaker that overlap the other speaker with little or no additional pausing from the other speaker at the point of the overlap. This simple rule is quite effective at identifying back-channels, back-offs and short recognitional overlaps. In some cases it may be helpful to further classify these interjections.
For a given language, the common lexical forms of back-channels can be enumerated. In US English for example these would include, but are not constrained to, continuers such as ‘uh huh’, ‘hmm’, ‘yeah’, ‘yes’, and ‘ok’. The phrases ‘thank you’, ‘thanks’, and ‘alright’ perform the dialog function of grounding or acknowledgment; These phrases function in a similar manner to back-channels in contexts that the automated process detects as an interjection. The phrase ‘oh’ indicates surprise but again functions like a back-channel when classified as an interjection.
According to other embodiments, a white list of words and phrases that are known to function as back-channels when detected as an interjection may be added to the process. For US English, this list might include the word and phrases mentioned above and could be extended or edited by someone skilled in the art. Other languages will have other equivalent sets of words or phrases or paralinguistic features which can be included if the speech-to-text engine supports them. If this list of words or phrases matches the phrase b1_text when the b1_interject rule is triggered or matches a2_text for the a2_interject rule then these interjections are considered to be back-channel interjections, as discussed elsewhere in this disclosure.
Examples of back-channel interjections detected by the automated process are shown in TABLE 8. In the table the interjection from one speaker is shown in curly braces embedded within the combined turn from the other speaker.
Any remaining interjections can be further classified by the process as restarted or continued back-offs. In some embodiments, restarted back-offs can be detected by the process by matching the text of the interjected utterance with the beginning of the next turn. In at least one embodiment, an interjection is classified by the process as a restarted back-off if there is an exact match between the interjection text and the left hand start of the following text from the same speaker with and without the first word removed.
TABLE 9 shows examples of interjections that are classified by the process as restarted back-offs. Interjections that are not classified as back-channels or restarted back-offs are classified as continued back-offs.
TABLE 10 shows examples of interjections that are classified by the process as continued back-offs. Sub-classification of these three different types of interjection enables different transforms to be performed by the process on the data and/or different visualization of the text to be automatically generated. For example, back-channels and restarted back-offs could be completely removed from the dialog text or hidden or de-emphasized in the visualization whereas continued back-offs could be moved to the start of the next turn following the same transformation pattern as an end_join.
Representing Phrases Using Non-Tabular Representations in General. The process described uses a method of operating on a tabular representation of phrases and re-structures this representation to implement the transformation. Other embodiments of the process may implement methods to gather words into phrases and, optionally, to gather phrases into larger phrases. Still other embodiments may incorporate or use other representation methods suggested by extrinsic sources available to those ordinarily skilled in the art. Such embodiments may utilize different structural representations of in addition to or alternative to organizing the dialog information in a linear table, while applying the same principles, methods and transforms according to the present invention to those different representations to achieve the objectives and benefits of the present invention.
For example, stand-off annotation such as the NXT XML format as disclosed by Calhoon (2010) may be integrated into the a process according to the present invention. Such formats, for example, separate the text of a transcription from annotated features of the conversation. The trigger rules and transform functions described by the process may be adapted to work with such a method. For example start and end pointers may be used by the process to represent the gathered phrases described in the method and these pointers may be transformed by the rules. Such a stand-off annotation format is well suited to further annotate the classification of interruption and overlap types.
Applying Text Processing Methods to Spoken Dialog. In addition to providing benefits for the visualization of spoken conversations, one or more of the embodiments according to the present invention may also be used to improve the performance of computer systems that analyze conversations and extract information from them. Examples of such systems would include conversational analytics platforms such as, but not limited to, Talk Discovery™ supplied by Discourse.AI Inc., also known as Talkmap, of Dallas, Texas.
Such state-of-the-art conversational analytics computing platforms often receive as input digital information including text of turns in a conversation labeled with the different speaker identities for each turn (utterance, phrase, etc.) in the conversation. Various processes are employed by these conversational analytics computing platforms to extract meaning or to classify emotion or intent from these conversations. Many such conversational analytics computing platforms have been trained or designed to work on written text such as transcripts of chat conversations on a web-site. In order to utilize the processes of the present invention on transcripts of spoken conversations, it is helpful, and often essential, to transform the spoken dialog into a form that closely resembles text-based conversations.
The embodiments of the present invention can, therefore, be used to enable advanced conversation analysis systems designed for or trained with text conversational data to be used effectively with spoken conversations without the need to train new models or design new processes.
Visualizations of Meaning and Emotion.
In at least one embodiment, the foregoing processes are implemented as an extensible framework into which additional sub-processes and transforms may be easily integrated and incorporated. Other methods for identifying turn boundaries and classifying the function of utterances in spoken conversations may be integrated into the solution process described in the foregoing paragraphs, such as but not limited to other processes available from the extrinsic art that make decisions based on text, timing and speaker identities alone.
In other embodiments according to the present invention, when the source audio digital recording is available to the process, additional signals such as the intensity and pitch of voice or the shortening or lengthening of words can be used by the process to the techniques described here to identify potential transition relevant places or classify competitive or cooperative interruptions.
As such, embodiments of the present invention are not limited by the method(s) used to discover the boundaries of utterances or categorize their function or group phrases together, nor are the limited by the transforms which are applied to the text.
The “hardware” portion of a computing platform typically includes one or more processors accompanied by, sometimes, specialized co-processors or accelerators, such as graphics accelerators, and by suitable computer readable memory devices (RAM, ROM, disk drives, removable memory cards, etc.). Depending on the computing platform, one or more network interfaces may be provided, as well as specialty interfaces for specific applications. If the computing platform is intended to interact with human users, it is provided with one or more user interface devices, such as display(s), keyboards, pointing devices, speakers, etc. And, each computing platform requires one or more power supplies (battery, AC mains, solar, etc.).
The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof, unless specifically stated otherwise.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Certain embodiments utilizing a microprocessor executing a logical process may also be realized through customized electronic circuitry performing the same logical process(es). The foregoing example embodiments do not define the extent or scope of the present invention, but instead are provided as illustrations of how to make and use at least one embodiment of the invention.
