Patent Application
20050246165
Research into the use of computers to understand what people communicate to one another, and how, has a long and deep history. Principally, the research has been conducted in the laboratories of large private and public corporations, governments, and universities. Progress has been made in such areas as linguistic analysis, non-verbal signaling, and speech recognition. Recent advances in the application of linked Hidden Markov Models (S. Basu, “Conversational Scene Analysis”, Ph.D. Thesis, MIT, September 2002), and, in particular, the application of such techniques as the “Influence Model” (C. Asavathiratham, “The Influence Model: A Tractable Representation for the Dynamics of Networked Markov Chains”, Ph.D. Thesis, MIT, October 2000), as applied to constructing the dynamics of interacting agents (T. Choudhury et al., “Learning Communities: Connectivity and Dynamics of Interacting Agents”, MIT Media Lab Technical Report TR#560, also in the Proceedings of the International Joint Conference on Neural Networks—Special Session on Autonomous Mental Development, July 2003), and Detrended Fluctuation Analysis (S. Basu, Ibid), have opened the field to new applications, which prior technologies were inadequately equipped to address.
A key advancement in this area is the application of quasi-syntactic analysis to verbal and non-verbal communication, which can yield insightful data without the burden of semantic determination of the content of an interaction. This work falls within the larger field of conversational scene analysis where prosodic cues are employed to identify an emotional state of an individual. Systems of this type have been assembled at institutions such as the Speech Technology and Research Laboratory at the Stanford Research Institute (SRI) and at the MIT Media Laboratory.
Commercial systems embodying various technologies seeking to determine emotional and/or semantic content have begun to appear on the market, for example, Utopy and Nemesyco. However, in the absence of syntactic and/or semantic voice content data, determining emotional states or stylistic non-content-based features of an interactive discourse to a reasonable accuracy is a hard problem; it requires a common-sense understanding of the discourse and an accurate application of context, and is a problem that has not lent itself well to computer automation.
To date, it has proven difficult to incorporate into a computer algorithm a human-like understanding of people-to-people communications of even the most elementary forms. The prior art has not solved the hard problem of common-sense reasoning, or assignment of the proper context to data streams obtained from daily exchanges of information among people.
Furthermore, the prior art does not provide a computerized system or method of using non-content-based cues to analyze a discourse, much less provide a means of conveying feedback to interacting participants in a discourse to move the discourse toward a desirable outcome. There is therefore a need for improved computerized methods of analyzing a discourse engaged in by interacting agents, such as conversing humans, the methods based at least partially on a combination of auditive and/or visual prosodic cues associated with the discourse.
The systems and methods described herein provide, in various ways, technologies related to discourse and/or behavior analysis in general, and conversational scene analysis in particular. In various embodiments, the systems and methods of the invention analyze a discourse based on prosodic cues, for example, and without limitation: spectral entropy, probabilistic pitch tracking, voicing segmentation, adaptive energy-based analysis, neural networks for determining appropriate thresholds, noisy autocorrelograms, and Viterbi algorithms for Hidden Markov Models, among others. Technologies that probe more deeply into the underlying structure of information in a human interaction show promise in enhancing the information, and may be used to supplement the analysis. For example, spectro-temporal response field functions for determining an individual's unique encoding of conversational speech (S. Basu, Ibid) may be employed to augment the conversational scene data collected from the audio and visual inputs of the systems and methods described herein.
The ability to measure styles of interaction among interacting individuals has many applications. These include, but are not limited to: teenagers wishing to improve their conversational image with one another; sales organizations hoping to improve their close rate with customers; and support personnel who wish to shorten the time of interaction with their clients while maintaining the quality of the support. Other applications include augmenting the types and amounts of information of real-time and non-real-time online social networking applications.
In one embodiment, the systems and methods described herein allow service providers to offer to subscribers quantitative and/or qualitative information aimed at helping determine the nature and effectiveness of communications among the subscribers and/or between the providers and the subscribers.
In an alternative embodiment, the systems and methods described herein provide the ability for customer sales and service departments to improve their operations and increase sales closing probabilities by giving them quantitative and/or qualitative information to facilitate determination of the nature of the communications between and among them and their customers. According to various practices, this information can be used for many other useful purposes to improve, or optimize, interactions, such as by reducing the amount of time spent in a conversation, improving the quality and/or flow of an interaction, or otherwise increasing the likelihood of a successful outcome or maintaining the interaction at a desirable state or within a range of states.
According to one practice, using a combination of a caller's name, phone number, zip code, and other indicia solicited, obtained, or inferred from the caller—for example, through an automated voice menu system prompting a caller to input certain relevant information—the nature of the call (request literature, open an account, etc.), and account information (if applicable), assumptions and inferences can be made about the caller's style of interaction, the context of the interaction, and one or more objectives of the interaction. Examples of contextual dependence of service rendering include sales and post-sales support; a caller requesting sales information about, for example, a computer that he or she may be interested in purchasing has needs that are ordinarily distinct from a customer who calls the manufacturer or an authorized dealer requesting repair or other post-sales service.
If a record exists of a previous call by the caller, then behavioral information associated with the record—such as, for example, information about an interactive style of the caller—might be available as a starting point, an initialization stage, for the systems and methods described herein. If no historical information is available about the individual caller, then according to one practice, the systems and methods disclosed herein refer to archived behavioral prototypes that most closely approximate the context and profile of the caller. The prototype information is then used, according to this practice, as a benchmark in evaluating and proceeding with the analysis of the caller's present interaction and/or guiding the discourse of the call in a desirable direction. The archived behavioral prototypes may be stored in a database accessible to a computer system implementing the methods according to the invention.
Actual and/or estimated caller information (perhaps obtained automatically from a database, or solicited from the caller through a sequence of menu-driven auditive and/or visual options and prompts), may then be used to match the caller to a service agent likely to have a productive interaction with the caller. According to one practice, when the caller calls, he or she is presented with a sequence of one or more menu options, during a subset of which the caller is prompted to enter relevant information; for example, the caller may be presented with an audio prompt as follows: “Please enter your account number,” or “Please enter your social security number.” As the call proceeds, the systems and methods described herein evaluate the call to determine whether it is likely to lead to a desired outcome for this type of call; the call-taker is advised on how to change the style, nature, or content of the interaction to move the conversation in a direction, or shift the conversation to a state, expected to increase the likelihood of a desired outcome. For example, the call-taker may be instructed to explain to the caller why the caller should open an IRA, make an additional IRA investment, purchase an annuity, etc. Although the embodiment above is described in terms of an incoming call, the systems and methods described herein work in substantially the same way in the context of an outgoing call.
According to one aspect, the systems and methods described herein provide a computerized method of analyzing a discourse engaged in by a plurality of interacting agents. The method includes the steps of measuring a first set of prosodic features associated with the discourse, during a first time interval; and at least partially based on the first set of features, determining a target set of prosodic features, wherein the target set is likelier to be associated with a target state of the discourse than the first set. According to one embodiment, the method includes suggesting to a subset of the agents, for example, by a feedback mechanism, a prosodic behavior for increasing a likelihood of producing the target state. In one embodiment, the method includes predicting a state of the discourse based at least partially on the first set of prosodic features; optionally, and based at least partially on the predicted state, the method includes suggesting to a subset of the agents a prosodic behavior for increasing a likelihood of producing the target state.
In one aspect, the systems and methods described herein include a computerized method of analyzing a discourse engaged in by a plurality of interacting agents, wherein the method includes the steps of: measuring a first set of prosodic features associated with the discourse, during a first time interval; and at least partially based on the first set of features, conveying to a subset of the agents a prosodic behavior for increasing a likelihood of producing a target state of the discourse.
In another aspect, the systems and methods described herein include a computerized method of analyzing a discourse engaged in by a plurality of interacting agents, the method comprising the steps of: measuring a first set of prosodic features associated with the discourse, during a first time interval; at least partially based on the first set of features, determining a first state of the discourse associated with the first set; and determining a change in the first set of features likely to incline the discourse away from the first state and toward a target state.
In yet another aspect, the systems and methods described herein include a computerized method of selecting a subset of agents to participate in a discourse, the method comprising the steps of: profiling a prosodic behavior of the agents based on at least one previous discourse engaged in by at least one of the agents; and based at least partially on the profiling, selecting the subset of the agents having an associated prosodic behavior likely to produce a target state of the discourse.
Further features and advantages of the invention will be apparent from the following description of illustrative embodiments, and from the claims.
The following figures depict certain illustrative embodiments of the invention in which like reference numerals refer to like elements. These depicted embodiments are to be understood as illustrative of the invention and not as limiting in any way.
To provide an overall understanding of the invention, certain illustrative practices and embodiments will now be described, including a method for analyzing a discourse engaged in by a plurality of interacting agents and a system for doing the same. The systems and methods described herein can be adapted, modified, and applied to other contexts; such other additions, modifications, and uses will not depart from the scope hereof.
In one aspect, the systems and methods disclosed herein are directed at improving interpersonal productivity and/or compatibility. According to one practice, the invention includes implementing conversational scene analysis on a computer having a processor, a memory, and one or more interfaces used for receiving data from, or sending data to, a number of interacting agents (typically, but not necessarily, humans) engaged in the discourse. According to this aspect, a system presents a result of the analysis to one or more interested parties—which may include one or more of the interacting agents—via a combination of a mobile phone, a personal digital assistant, and another device configured for such purpose, and enabled with a combination of voice (e.g., a speaker or other audio outlet), tactile (a vibration mechanism, as in a mobile phone), visual (e.g., a web browser or other screen), and other interfaces.
Optionally, the system according to this aspect conveys feedback to a subset of the agents, the feedback being directed at altering a behavior of the subset of the agents, thereby inclining the discourse away from an undesirable outcome, toward a desirable outcome, maintaining a status quo, or a combination thereof. The feedback may be conveyed to the subset of the agents in a number of ways: auditive feedback, visual feedback, tactile feedback, olfactory feedback, gustatory feedback, synthetically-generated feedback (such as, for example, a computerized message or prompt), mechanical or other physical feedback, electrical feedback, a generally sensory feedback (such as, for example, a feedback that may stimulate a biometric characteristic of an agent), and a combination thereof.
In one aspect, the systems and methods described herein extend and implement these and other concepts for application to practical everyday settings of commercial and consumer use.
According to a typical practice, the interaction characterizing the discourse 103 is predominantly speech-based. An example of this is when two people 101 and 102 converse using mobile phones, internet voice-chat software, or other media, without seeing each other. There are, however, other exemplary practices wherein the discourse includes not only speech, but also a non-verbal communication modality, such as, for example, and without limitation, speech accompanied by a combination of visual cues associated with posture and/or gesture.
Exemplary prosodic features that may be employed, and examples of what those features may imply in terms of human behavior, by the systems and methods described herein are tabulated in Tables 1A-1I, 2, and 3 below. The tabulated lists are not intended to be comprehensive or limiting in any way. Other prosodic features not listed may be employed by the systems and methods disclosed herein, without departing from the scope hereof.
These and other prosodic auditive and visual cues are described in, for example, and without limitation: “The Profiling and Behaviour of a Liar,” by John Boyd, Manager Corruption Prevention, Criminal Justice Commission, Queensland, Australia, presented at SOPAC 2000, Institute for Internal Auditors—Australia, South Pacific, and Asia Conference, 27-29 Mar., 2000; “Silent Messages, “by A. Mehrabian, Wadsworth Pub. Co., 1971; and “Emotion Recognition in Human-Computer Interaction,” IEEE Signal Processing Magazine, Jan. 2001, pp. 32-80.
Prosodic cues such as those listed in Tables 1A-1I, 2, and 3 may be employed by the systems and methods described herein to analyze an exemplary discourse 103 engaged in by the agents 101 and 102 interacting with each other using a videoconferencing system, or using mobile communication devices configured to capture image and/or video data in conjunction with audio information. In yet another illustrative embodiment, the discourse 103 is substantially non-speech-based, such as, for example, when the two agents 101 and 102 converse using a text-based Internet chat software, such as an “instant messaging” application. According to one practice, the agents 101 and 102 use a combination of emoticons, graphical icons, exclamation marks, or various keyboard characters as prosodic signals to express or convey a tone, attitude, or interactive style in their computerized communication; these prosodic cues generally augment and accompany syntactic and semantic content associated with the discourse.
Style includes such parameters as how fast an agent talks, how long the empty spaces are between utterances of the speakers, average length of speech by each speaker, etc. These parameters can be used for assessing a characteristic of the interaction, for example, trust, liveliness, or other characteristics that develop among the participants in the discourse. According to one aspect of this practice, the systems and methods described herein extract prosodic cues associated with the discourse, such as, and without limitation, typing rate, use of iconic visuals expressing a mental state or tone, capitalizations or exclamation marks in the text, pause length between responses, telemetric measurements in general, or biometric measurements in particular, of the agents 101 and 102, and other non-syntactic, non-semantic features of the discourse generally classified as prosodic. Although typically the analysis is performed substantially in real time, this is not necessary. According to one practice, the analysis is performed post hoc, from a record of the discourse. For example, the interaction may be through a set of e-mail exchanges between the agents 101 and 102, saved and archived. Alternatively, the analysis may be performed on an audio, video, or audiovisual recording of the discourse.
Typically, prosody includes features that do not determine what people say, but rather how they say it. Traditionally, the term has referred to verbal prosody, that is, the set of suprasegmental features of speech, such as stress, pitch, contour, juncture, intonation (melody), rhythm, tempo, loudness, voice quality (smooth, coarse, shaky, creaky phonation, grumbly, etc.), utterance rate, turn-taking, silence/pause intervals, and other non-syntactic, non-semantic features that are generally embedded in a speech waveform and typically accompany vowels and consonants in an utterance. Recently, however, the definition has been broadened to include visual prosody, that is, specific forms of body language that interacting agents employ to communicate with one another during their discourse; examples of visual prosody include, without limitation, facial expressions such as smiling, eyebrow movement, blinking rate, eye movement, nodding or other affirmative or dismissive head movements, limb and other bodily gestures, such as strumming or tapping a finger, folding of arms, shrugging, tapping of feet, adjusting clothing, fidgeting, and various other forms of communication generally classified as kinesics and proxemics, etc., at least partially listed in Tables 1A-1I, 2, and 3. Herein, prosody is used in its broader scope, and includes a combination of verbal (more generally, auditive) and visual features.
In one embodiment, the discourse may be substantially visual, and may have insubstantial speech or other auditive content. Instant messaging between two interacting humans who do not see each other is an example of this embodiment.
If the discourse 103 includes speech, as it would in a typical embodiment, then a speaker separation (otherwise known as a source separation) method may be applied to the data signal 123 to distinguish information associated with the speaker/agent 101 from data associated with the speaker/agent 102. For example, independent component analysis, principal component analysis, periodic component analysis, or other source separation methods may be used to separate data associated with the agents 101 and 102. According to one practice, a hidden Markov model (HMM) may be employed to separate speech waveforms associated with various speakers (and optionally from ambient sounds) using a low-frequency energy-based scheme (T. Choudhury and A. Pentland, “Modeling Face-to-Face Communication Using the Sociometer”, Proceedings of the International Conference on Ubiquitous Computing, Seattle, Wash., October 2003).
In one practice, a subset of the data signals 121-123 may include noise, and one or more noise-removal methods may be used to separate, or filter, the noise to substantially suppress it or to otherwise alter its form. Signal source separation used by certain embodiments of the systems and methods described herein follow principles described in the following exemplary reference, among others: “Unsupervised Adaptive Filtering, Volume 1: Blind Source Separation”, by Simon Haykin (Ed.), Wiley-lnterscience, 2000, ISBN 0471294128.
The data signals 121-123, which generally contain a combination of auditive and visual data, may be obtained using a variety of methods. For example, auditive data may be obtained using microphones present near one or both of the agents 101 and 102.
The information collected from a combination of the data signals 121-123 is fed to an input processor 130 associated with a computer system 150. According to one practice, the computer system 150 includes various components: the input processor 130, the output interfaces 140a and 140b, the memory 160, the CPU 170, and the support circuitry 180. The CPU 170 serves as the data processing engine implementing the methods according to the invention; the support circuitry 180 provides various services to the computer system, such as supplying and regulating power to the computer 150; and the memory 160 provides data storage for the computer 150, and typically includes both persistent and volatile memory. The memory 160 includes software configured to execute on the computer 150 to implement the methods of the invention, such as, for example, the prosodic feature extraction algorithms 162 and the flow of interaction analysis algorithms 164. Other software applications that may be needed or desirable in a particular embodiment are not shown in the figure, but it is understood that the computer memory 160 contains such software accordingly. The various links 163, 165, 167(a-b), 169, 182, and 184 denote communications that can occur between the various respective components of the computer system 150. For example, the link 163 shows an optional connection between the input processor 163 and the memory 160, enabling the processor and the memory to exchange information. The embodiment depicted by
An embodiment according to
Alternatively, or additionally, the behavior of the agents 201 and 202 in the current discourse may be compared with a public archive of behaviors of representative agents. For example, the archive 294, according to one embodiment, includes information about other agents who have engaged in a similar discourse (where by similar discourse it is meant that the discourse is conducted under a similar context, perhaps having a similar outcome, e.g., closing a sale). In this embodiment, prosodic features associated with the archived discourses 294 are available. According to one practice, the prosodic features extracted by the feature extractor 262 from the discourse 203 are analyzed, compared with, and/or mapped 272 to the archived features in 294. Accordingly, the feedback 231 and/or 232 is rendered to the respective agents 201 and 202, via the output interface 240 of the computer system 150 (not shown in
In one practice, information stored in the archives 292 and/or 294 may be used by the systems and methods of the invention to predict a future behavior of one or more of the agents 201 and 202, and/or a future state (such as a future characteristic) of the discourse 203. In one exemplary aspect, a vector of prosodic cues is measured from the discourse 203 and compared against statistical information stored in one or both of the archives 292 and 294. According to the statistical information, the likelihood of a future characteristic of the discourse and/or a future action of one or more of the agents 201 and 202 is assessed.
For example, statistical information may indicate that given the current measured vector of prosodic cues, the likelihood of a shouting match ensuing is high; therefore, one or both of the agents 201 and 202 may be given feedback suggesting to them to lower their voices or to modify another set of one or more prosodic features to steer the discourse away from the predicted shouting match. Alternatively, or additionally, the systems and methods disclosed herein may force a set of one or more constraints on the discourse in anticipation of the predicted state; for example, if the discourse 203 includes a telephone conversation between the agents 201 and 202, the systems and methods described herein may—in anticipation of a shouting match ensuing—lower the volume of one or both speakers (possibly even without their consent), thereby potentially preventing a breakdown in the discourse (an undesirable outcome).
According to another exemplary aspect, a state vector including, for example, the vector of prosodic cues, is constructed and measured at predetermined time instants of the discourse. A Kalman filter is then used to process past and current information, based on a mathematical (such as a Bayesian) model of the discourse to predict a subsequent state. Recursive filters other than the Kalman filter may be used in estimating the vector of prosodic cues. Alternatively, the prosodic features may be divided into various subsets, each subset being estimated by a method specifically tailored or otherwise suitable for that subset. For example, one subset of the prosodic cues may be processed using a Kalman filter, and another subset may be processed using another type of filter, or even a nonlinear filter. In any event, based on the predicted discourse state or characteristic (including, for example, agent behavior), the systems and methods described herein can render feedback 231-232 to a subset of the agents.
According to various aspects, at least one of the data signals 321-322, 324-325, and 343 may be available, not necessarily all of them. Moreover, the availability of the data signals may be time dependent; for example, whereas a data signal may be available for a particular first time interval, it may not be for at least a portion of a second time interval distinct from the first time interval. This can happen, for example, if the number of the agents changes during the discourse, wherein one or more new agents enter the discourse and one or more agents leave (this is typical of an Internet chat room setting). According to one embodiment, the first and second time intervals do not overlap; one of the two intervals is in the future, relative to the other. In another embodiment, the first and second time intervals at least partially overlap, but remain distinct based on having distinct temporal boundaries; in this embodiment, at least a portion of one of the two intervals is in the future and/or in the past relative to at least a portion of the other time interval.
According to one embodiment, the discourse of
As mentioned above, even the number or make-up of the agents may change during a discourse. While some agents may partake in a negotiation for purely administrative and/or formal reasons, other agents may engage in the negotiation as leading advocates of their points of view, and as such may deliberately and/or competitively seek to influence other agents representing alternative bargaining positions.
A desirable outcome in one phase of the discourse is not necessarily as desirable (or even desirable at all) in another phase. Agents may also participate in the discourse for the full duration of the discourse, or they may participate temporarily or intermittently.
Behavioral dynamics of agents in a dyadic discourse (a discourse involving primarily two competing interests) are typically distinct from the behavioral dynamics of agents in a multilateral discourse where multiple competing interests are at play; for example, it has been observed that it is easier to convince or otherwise desirably influence other, competing agents when the there is primarily one competing bargaining position, than it is to convince other agents in a multilateral setting where there are generally multiple competing, and possibly even conflicting, interests.
According to various embodiments, the systems and methods described herein are directed at accounting for various phases of the discourse, their corresponding desirable outcomes, and the behavioral dynamics of the agents during those phases. Accordingly, when the agents 301-305 represent multiple competing positions/interests in the multilateral, perhaps negotiation-based, discourse 310, the systems and methods described herein adjust a subset of the feedbacks 331-332, 334-335, and 343 to at least account for the phase of the discourse at a time when feedback is rendered. The adjustment is based at least partially on a dynamic database of public and private style prototypes (not shown in
To avoid clutter in
The interaction style comparator 472 produces a characterization 490 of a behavioral difference; in one embodiment, the characterization 490 of the comparisons shows higher level stylistic patterns suggesting particular modifications, such as slowing down, speeding up, reducing volume, changing intonations and/or body language, etc., which can lead, for example, to better trust and synchrony between the agents 401 and 402. Alternatively, the modifications may be recommended at least in part because in similar situations they have frequently resulted in a desired outcome. The behavioral difference 490 may include a difference between the behaviors of the agents 401 and 402. Alternatively, or additionally, the behavioral difference 490 may include a difference between a style mapping associated with the agent 401 and a style mapping associated with the agent 402, possibly indicating that the agents have incompatible styles or complementary styles. In any event, the characterized behavioral difference 490 is then used to produce a set of one or more behavioral modification suggestions 495 to be conveyed via one or both of the feedback paths 431 and 432 to the respective agents 401 and 402.
In a typical embodiment, the suggested behavioral modifications take into account the context of the discourse and acceptable behavioral norms 491, or norms of behavior that are calibrated according to, or that are otherwise applicable to, the context of the discourse between the agents 401 and 402. After all, a behavioral modification suggestion that may be appropriate in the context of a sales transaction may not be appropriate in the context of a police-suspect interrogation, for example.
The error term 540 is produced by taking a difference of the desired interaction 570 and the interaction 530 being analyzed. The difference may be associated with a state of the desired interaction 570 and a state of the interaction 530 being analyzed. Alternatively, the difference may be associated with a set or vector of measurable features characteristic of the desired interaction 570 and a corresponding set or vector associated with the subject interaction 530. Based at least in part on the error 540, a next action, state, or characteristic of the interaction or behavior of the agent 501 is predicted by the model 580. The prediction model 580 may optionally employ a behavioral archive 520 (containing a combination of public norms and private styles of behavior, as described in relation to the previous figures) to predict the next action in the current discourse.
Alternatively, or additionally, the predictive model 580 may base its output at least in part on a hidden Markov model and/or influence model representation 510 of the discourse and/or a subset of the interacting agents. For example, by knowing the influence that the agent 501 has on another agent, and vice versa, the predictive model may at least partially predict a next state or action by the agent 501, or by the other agent in the discourse. In one practice, the influence of the agent 501 on another set of one or more agents (not shown in
A variety of measures of centrality, for example, those known widely in the social network theory, may be used by the systems and methods described herein, depending on the context. According to one embodiment, centrality includes betweenness centrality, which measures how much control an individual/node/agent in a social network has over the interaction of other individuals belonging to the network who are not directly connected to each other. In one aspect, betweenness centrality captures the role of “brokers” or “bridges” in a network, those possessed of large indirect ties and capable of connecting or disconnecting portions of the network.
According to one embodiment, closeness centrality—which, on a graph representation of a social network, is the sum of geodesic distances of an agent (i.e., node) to all other agents (nodes) belonging to the network—is used. In an alternative embodiment, eigenvector centrality—which is a measure of walks of all lengths, weighted inversely by length, emanating from a node in a mathematical graph representing a network of interacting agents—is used. In one embodiment, degree centrality is used; this measure of centrality is associated with the total number (or weight) of ties that an agent (or node in a network) has with all other agents. In one practice, expansiveness and/or popularity of an agent may be inferred from the agent's degree centrality. An agent with a relatively large degree centrality is typically considered to be a connector or a hub. In some embodiments, one or more variants of these measures of centrality may be used, for example, relative degree centrality (ratio of the degree of an agent over the highest degree of any agent in the network), relative betweenness centrality, and relative closeness centrality.
In addition to, or instead of, one or more agent-based measures of centrality, the systems and methods described herein may use one or more network-wide measures of centrality, for example, network degree centralization, network closeness centralization, network betweenness centralization, etc. A network centrality measure is considered useful in assessing a characteristic of a network of interacting agents, because, loosely speaking, the larger the centrality measure of a network, the higher the network's cohesion, and, generally, the higher the likelihood of having the agents belonging to the network reaching a common goal. A more cohesive network also typically results in better network-wide control and/or influence over its individual member agents.
In one embodiment, the graph representation is a directed graph, with a directed arc pointing away from a node representing the agent 501 denoting an influence or control that the agent 501 has on another agent to whom (or to which) the arc points. An out-degree measure associated with the node representing the agent 501 may be indicative of the power, prestige, control, respect, or other analogous hallmark of influence that the agent 501 wields with respect to the other agents engaged in the discourse. If the node associated with the agent 501 has a relatively high out-degree, then a degree centrality of the agent 501 is high, thereby indicating that the agent wields considerable influence. Accordingly, a future state or characteristic of the discourse is determined by taking into account the degree centrality of the agent 501.
A directed arc pointing into the node representing the agent 501 may denote support that the agent receives from another node representative of another agent from whom (or from which) the arc originates. Alternatively, a directed arc pointing into the node representing the agent 501 may be indicative of a level of influence, power, control that the agent 501 is under, with respect to another agent from the representative node of whom (or which) the arc emanates. In one exemplary embodiment, an in-degree measure of the agent 501 indicates support, such as by voters, in the discourse. In another embodiment, it may indicate the subservience of the agent, if the in-degree is indicative of the influence that another agent has on the agent 501.
In another aspect, the systems and methods described herein employ the influence model of Asavathiratham, as described in, for example, “Learning Communities: Connectivity and Dynamics of Interacting Agents,” by T. Choudhury et al., MIT Media Lab Technical Report TR#560, which also appeared in the Proceedings of the International Joint Conference on Neural Networks—Special Session on Autonomous Mental Development, 20-24 Jul. 2003, Doubletree Hotel, Jantzen Beach, Portland Oregon—Special Session W3S: Autonomous Mental Development, Wednesday, July 23, 2:40 PM , “Learning Communities: Connectivity and dynamics of interacting agents” [#854], Tanzeem Choudhury, Brian Clarkson, Sumit Basu, and Alex Pentland, MIT.
According to various embodiments, the actual output 590, the current interaction 530, the desired interaction 570, and the error 540 may include a vector representation of prosodic features associated with the discourse. According to one practice, the error 540 includes a Euclidean difference between the vector representative of the subject interaction 530 being analyzed and the vector representing the desired interaction 570. Alternatively, a Euclidean distance between the current interaction vector 530 and the desired interaction 570 may be used to characterize the error 540.
The inverse model 550 typically includes a mapping between a set of parameters characteristic of the desired interaction 570 and the set of behaviors that bring about the desired interaction. For example, the inverse model may map the desired outcome of enabling a 911 operator (agent 501) to assist a frantic caller (not shown) to a certain voice volume/rate profile; that is, if the operator 501 has a voice volume within a prescribed range and/or speaking rate within a prescribed range, then a desired interaction 570 is likely to ensue. The inverse model 550, then, is used by the systems and methods described herein to impact the behavioral modification suggestions 560 formulated to provide feedback to the agent 501. Based on the predicted state/action/characteristic and on the output of the inverse model 550, one or more behavioral modification suggestions 560 are conveyed to the agent 501, aimed at bringing the current interaction 530 closer to the desired interaction 570.
Optionally, the model shown in
In one embodiment wherein the current interaction 530, the error 540, and the desired interaction 570 are Euclidean vectors of prosodic features, the predictive model 580 includes a Kalman filter that predicts a next state of the discourse based on the current and past states of the discourse, using, for example and without limitation, Bayesian information and optimization criteria. Therefore, if the discourse is divided into feedback iteration cycles, the Kalman filter uses the current state of the discourse and the past states (at previous feedback cycles), to predict the state at the next cycle.
The systems and methods described herein employ, in various embodiments, principles of recursive filtering, including Kalman filtering, to predict future states of a time-evolutionary process, such as an evolving discourse engaged in by a plurality of interacting agents. Recursive filtering principles include those described by the following exemplary references: “Fundamentals of Adaptive Filtering”, by Ali H. Sayed, John Wiley and Sons, 2003, ISBN 0471461261; “Kalman Filtering and Neural Networks”, by Simon Haykin, Wiley-lnterscience, 2001, ISBN 0471369985; “Linear Estimation”, by Thomas Kailath et al., Prentice-Hall, 2000, ISBN 0130224642; and “Adaptive Filter Theory, 4th Edition”, by Simon Haykin, Prentice-Hall, 2001, ISBN 0130901261.
As mentioned earlier, the inverse model 550 that produces one or more output controls to effect a change in the interaction is substantially a functional mapping taking prosodic features as inputs and producing behavioral actions (including, but not limited to, prosodic modifications) as outputs. Various models can be used to characterize the inverse model 550. For example, and without limitation, stochastic Bayesian network models that employ asymptotic approximations, maximum likelihood estimation (MLE), including, for example, an expectation-maximization (EM) implementation of the MLE, or algorithms that use neural networks and/or radial basis function networks to model the stylistic variables of interest to the systems and methods described herein may be used in various embodiments.
In certain embodiments, approaching the desired interaction involves simultaneous optimization of multiple objectives. Using single-objective optimization procedures, arriving at a solution (whereby a target, desired interaction is specified) may be difficult. For such embodiments, evolutionary algorithms may be employed to find a Pareto-optimal set of features characterizing a desired interaction. In particular, a genetic algorithm may be used to iteratively home in on a Pareto-optimal boundary descriptive of the desired interaction. Accordingly, one or more of the agents engaged in the discourse are given instructions or suggestions on how to modify their respective behaviors to drive the discourse to a point on the Pareto-optimal boundary of solutions. Methods of evolutionary algorithms in general, and genetic algorithms in particular, are described in “Multi-Objective Optimization Using Evolutionary Algorithms,” by Kalyanmoy Deb, John Wiley & Sons, 2001, ISBN: 047187339X.
According to one embodiment, the flow of the methods and systems described herein is as follows. As a first step, determine whether the systems and methods described herein will be initially customized by training based on individual agents or sets of agents within a particular context (e.g., conversing Japanese school girls). Next, determine whether the systems and methods described herein rely on global human-communications protocols.
When initializing the systems and methods described herein, using as an optional input to the system a set of desired outcome parameters (e.g., time to obtain x % compatibility among persons A, B, and C; degree of turn taking, dominance, % air time, % shakiness of voice, % synchrony, speed of speaking and/or non-verbal gesturing, etc.) is specified by one or more agents or by another party. The system is then trained to develop prototype patterns for the individual and/or an ideal utopian pattern of interaction, wherein ideal is context dependent, for example, business or pleasure.
As a next, optional, step, post-training information is gathered from a set of two or more agents to find matches among those who are compatible in accordance with a specified compatibility algorithm. For example, agents may be sought to engage in a discourse, based on archived normative behaviors of eigenagents, and their corresponding behavioral prosodic features. Data from a new set of agents may be collected and compared with the archived data, to determine which subset of the new agents most closely, or sufficiently closely, meets a compatibility measure.
A subsequent step of an embodiment of the systems and methods described herein includes providing feedback to a subset of the agents to allow them the opportunity to modify their behavior. The feedback may optionally include providing to the subset of the agents updated information about the interactions (such as measured prosodic cues). The interacting agents may use the feedback to effect changes in their behaviors.
As a next step, an embodiment of the systems and methods described herein includes calculating the various agents' inputs and determining clusters of behaviors that maximize the likelihood of a desired outcome. At prescribed intervals, the systems and methods described herein optionally update the global normative behavior archives and/or the agent-specific behavioral archives, for future use.
Data classification and pattern analysis techniques, used by various embodiments of the systems and methods described herein, follow principles laid out in the following exemplary reference, among others: “Pattern Classification: 2nd Edition”, Richard O. Duda et al., Wiley-lnterscience, 2000, ISBN 0471056693. Collective/public behavioral prototypes or individual agent-specific behavioral prototypes that are used by the systems and methods described herein as archived databases for matching/mapping a current interaction to normative interactions, can be constructed using principles known in data classification, pattern analysis, and estimation theory.
One method of constructing an archive of normed (prototypical) collective behavior, for example, is to select prosodic features of interest and measure those features for a number of groups of agents in similar interactive contexts. Multivariate probability density or mass functions can be constructed based on the data, using, for example, multivariate histograms of historical measurements of the prosodic cues in similar contexts. Other methods may be employed to construct probabilistic models of the prosodic features associated with various types, states, or characteristics of discourses. Models of behavioral dynamics may be used to construct statistical models of agent behavior.
As mentioned above, one way of looking at the prosodic features is by constructing a vector of measured prosodic cues. A multivariate probability density (or mass) function may then be constructed based on measurements of the prosodic cues vector. The probabilistic model may be updated as new measurements of the prosodic cues are made.
Alternatively, or additionally, if a known probability density function is considered to model the normative behavioral data reasonably well, a combination of one or more estimation techniques may be used to determine the parameters specifying the particular form of the probability density function. For example, if in a particular embodiment, a multivariate Gaussian density function is considered to be a reasonable model of the normative behaviors of the eigenagents in a particular context, then the parameters (such as the mean vector and covariance matrix) associated with the multivariate Gaussian density function may be estimated from the collected data using known statistical techniques. Once new measurements are made from the subject discourse being analyzed, methods such as maximum likelihood may be used to estimate a state and/or characteristic of the discourse.
The contents of all references, patents, and published patent applications cited throughout this specification are hereby incorporated by reference in entirety.
Many equivalents to the specific embodiments of the invention and the specific methods and practices associated with the systems and methods described herein exist. Accordingly, the invention is not to be limited to the embodiments, methods, and practices disclosed herein, but is to be understood from the following claims, which are to be interpreted as broadly as allowed under the law.
This application incorporates by reference in entirety, and claims priority to and benefit of, U.S. Provisional Patent Application No. 60/566,482, filed on 29 Apr. 2004.
| Number | Date | Country | |
|---|---|---|---|
| 60566482 | Apr 2004 | US |
