The present disclosure relates to the field of automated data processing, and more specifically to the application of ontology programming to process and analyze communication data. In the realms of computer and software sciences and information science, an ontology is a structural framework for organizing information regarding knowledge and linguistics within a domain. The ontology represents knowledge within a domain as a hierarchical set of concepts, and the relationships between those concepts, using a shared vocabulary to denote the types, properties, and interrelationship of those concepts. For example, the ontology models the specific meanings of terms as they apply to that domain.
Methods are disclosed herein for expanding an initial ontology via processing of communication data, wherein the initial ontology is a structural representation of language elements comprising a set of entities, a set of terms, a set of term-entity associations, a set of entity-association rules, a set of abstract relations, and a set of relation instances. An exemplary method includes providing the initial ontology, providing a training set of communication data, processing the training set of communication data to extract significant phrases and significant phrase pairs from within the training set of communication data, creating new abstract relations based on the significant phrase pairs, creating new relation instances that correspond to the significant term pairs, storing the significant phrases as ontology terms ontology and associating an entity for the added terms, and storing the new relation instances and new abstract relations to the initial ontology.
Also disclosed herein is a method for extracting a set of significant phrases and a set of significant phrase co-occurrences from an input set of documents. An exemplary method includes providing a generic language model and providing the set of documents. The exemplary method extracts a set of significant phrases by, for example, generating a source-specific language model by subdividing each document into meaning units, accumulating phrase candidates by creating a set of candidates where each candidate is an n-gram and iterating over the n-grams to compute a prominence score for each n-gram and a stickiness core, and filtering the candidate phrases by calculating a frequency for each of the candidate phrases and calculating an overall phrase score for each of the candidate phrases. The exemplary method can extract significant phrase co-occurrences by, for example, iterating over the meaning units and locating the occurrences of individual phrases, counting the number of co-occurrences of pairs of phrases in the same meaning unit, computing a probability of a phrase and a probability of the co-occurrence of a pair of phrases based on the count, calculating a log-likelihood of the co-occurrence using the probability of the phrase and the probability of the co-occurrence of a pair of phrases and identifying a significant co-occurrence of the pair of phrases if the log-likelihood is over a predetermined log-likelihood threshold.
The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description, drawings, and from the claims.
According to the present invention, an ontology may be developed and applied across all types of communication data, for example, all types of customer interactions (which may include interactions in multiple languages) as a tool for processing and interpreting such data. The communication data may document or relate to any type of communication, including communications made via phone, via email, via internet chat, via text messaging, etc. For example, communication data may contain any spoken content or any written correspondence or communication, including but not limited to live speech, audio recording, streaming audio, transcribed textual transcripts, or documents containing written communications, such as manuscripts, web pages, email, physical mail, text messages, chats, etc. In the exemplary context of a customer service application, the communication data may be between a customer service agent or an automated system, such as an interactive voice response (IVR) recording, and a customer or caller. While the present disclosure is often exemplified herein by describing an embodiment involving the analysis of audio data, such as recorded audio transcripts, it is to be understood that in alternative embodiments other forms of oral or written communications may be used or analyzed. A particular ontology may be defined for and applied to any domain, and other examples include financial services, consumer products, subscription services, or some other business application involving communication data interactions.
In the generation, refinement, or development of an ontology, repeating patterns are identified and ranked based upon statistical significances and then clustered into terms and term relationships. The disclosed solution uses machine learning-based methods to improve the knowledge extraction process in a specific domain or business environment. By formulizing a specific company's internal knowledge and terminology, the ontology programming accounts for linguistic meaning to surface relevant and important content for analysis. For example, the disclosed ontology programming adapts to the language used in a specific domain, including linguistic patterns and properties, such as word order, relationships between terms, and syntactical variations. Based on the self-training mechanism developed by the inventors, the ontology programming automatically trains itself to understand the business environment by processing and analyzing a corpus of communication data.
The disclosed ontology programming, once built and refined for a specific business application, is applied to process communication data to provide valuable analytics for a variety of business needs. For example, the ontology programming can then be utilized to detect and surface meaningful items in a data set, such as a database of recorded employee-customer interactions, and can mine the data set to extract and analyze business data based on an enhanced formulization of a company's internal knowledge and terminology.
An exemplary embodiment of the presently disclosed ontology solution incorporates three main stages. As seen in
Generally, an ontology as O as disclosed herein can be defined as , , , , , , wherein is a set of entities, is a set of terms, is a set of term-entity associations, is a set of entity-association rules, in which, is a set of abstract relations, and is a set of relation instances. Terms are individual words or short phrases that represent the basic units or concepts that might come up in the communication data. Thus a set of terms T can be defined as a word n-gram that has some meaning. Non-limiting examples of terms, as used herein, include “device”, “iPhone”, “iPhone four”, “invoice”, “I”, “she”, “bill”, “cancel”, “upgrade”, “activate”, “broken”, or “cell phone”, “customer care”, or “credit card.” However, these are not intended to be limiting in any manner and are merely exemplary of basic units or concepts that may be found in a customer service interaction. In certain embodiments, all words in the corpus, or set of communication data, can only be associated with one term, and each term can only be counted once.
Development of an ontology involves the identification of term candidates. A set of communication data used for training purposes is divided into potential terms, or term candidates. Terms are then selected from those term candidates. Strong term candidates contain words or word sets that are compact and, in the instance of word sets, the frequency of finding the word set together is very high. An example of a term containing a word set is “credit card number,” as those words very often appear together and refer to a particular, defined object. In addition, good terms often contain words that make more conceptual sense when they are together, as opposed to being on their own. For example, the term “Nova Scotia” is comprised of words that make sense when found together, and would likely not appear or make sense separately.
The frequency that the words of a particular word set, or term, appear together may be referred to as the “stickiness” of the term. A “sticky” term is one whose words appear frequently appear together in the corpus. The higher the stickiness ranking, the stronger the term, as it means that the term has meaning in the corpus as a concept. Salient terms are those that stand out or have a higher score relative to similar or neighboring terms. Non-salient terms and less-salient terms are those that appear many times or a relatively large number of times in many different contexts. The score of such non-salient or less-salient terms is lowered as compared to the score for salient terms. The logic is that salient terms are likely to be meaningful as a concept, whereas non-salient terms are not likely to express a particular concept. For example, the score of the term “account number” would be higher than the score of the term “the account number” because the word “the” appears in many different contexts and also by itself. Therefore, the word “the” does not add any significant meaning when joined with the term “account number.”
Entities are broader concepts that encapsulate or classify a set of terms. Entities describe semantic concepts to which classified terms are related. Non-limiting examples of classes, may include “objects”, “actions”, “modifiers”, “documents”, “service”, “customers”, or “locations”. However, these are not intended to be limiting on the types of entities, particularly the types of entities that may appear in an ontology directed to a specific or specialized domain. Thus a set of entities can be organized in a hierarchical tree-like structure, where for each entity E∈, let π(E) be the parent entity of E.
The set of entities and terms in the ontology are connected by a set of term-entity associations ⊆× governed by entity-association rules ⊆××(∪{{tilde over (0)},{tilde over (1)},{tilde over (2)}}). Each term in the ontology is associated with at least one entity, namely ∀T ∃E T,E∈. In some embodiments it is possible for a term to have multiple entity associations. For example, if the term is “jaguar”, the term may be associate with the entity “Animal” and with the entity “CarBrand.” The distance between a term T and E, denoted d(T,E) is 0 in case T,E∈. Alternatively, if there exists an entity E′ such that T,E′∈and E is an ancestor of E′, then d(T,E)=d(E′,E); in either of these two cases, TE. Otherwise, d(T,E)=∞.
Abstract relations express relations between two ontology entities. A set of abstract relations can be defined as ⊆×. The distance between two entities E1,E2∈ in the hierarchy chain, denoted d(E1,E2), can be defined as the number of steps down or up the hierarchy of entities. For example, if E1 is an ancestor of E2 then d(E1,E2) is defined as the number of steps down the hierarchy, whereas if E2 is an ancestry of E1 then d(E1,E2) is defined as the number of steps up the hierarchy of entities. If none of these conditions apply however, then d(E1,E2)=∞.
Relation instances express relations between ontology terms. A set of relation instances can be defined as ⊆×. For example, the term “pay” may be related to the term “bill” to form the relation “pay> bill.” In another non-limiting example, the term “pay” may be associated under the entity “action” and the term “bill” may be defined in the entity “documents”. Still further binary directed relationships can be defined between these entity/term pairs. For example, the action/pay pair may be related to the document/bill pair in that the payment action requires an underlying document, which may be a bill. In another non-limiting example, the term “broken” may be defined in the entity “problems” and the term “iPhone” may be defined in the entity “device”. The problem/broken pair can also have a directed relationship to the “devices” entity in which the “iPhone” term is a specific example as represented by the devices/iPhone pair.
As exemplified in
A canned ontology can be developed in various ways. For example, a canned ontology may be developed by taking data samples generated by multiple different users or classes in a particular industry. Alternatively, a canned ontology may be created by combining multiple ontologies developed using sample data sets from a particular industry. For example, multiple users may develop an ontology for their particular business based on their own internal data. Those individual ontologies may then be combined through a process of comparison, wherein the common elements in the ontologies receive heavier weight than the elements that differ. In still other embodiments, a canned ontology could be developed over a series of training processes where one user develops an ontology based on its data, and then the next user uses the first user's ontology as a canned ontology input to its training process. Thereby, each subsequent user implements a previous user's output ontology as a canned ontology 201 input, and amends or refines that canned ontology through the training process to develop its own ontology.
In
In certain of the embodiments, the ontology training phase 1 is not executed until a certain, predefined amount of data is gathered for the training. In one embodiment, a configured scheduler may monitor the data gathering process and count the number of records or amount of data added. When the number of records or amount of data in the training data set 205 reaches that predetermined amount, the scheduler may execute the ontology training process 1. Alternatively or additionally, the scheduler may monitor the types and/or variety of data added to the training data set 205 so as to ensure that the training 301 does not begin until certain types and/or varieties of data are available to the training set. In certain embodiments, the communication data is transformed into a usable format as part of training phase 1. For example, audio data from one or more customer interactions between a customer service agent/IVR and a customer/caller can be automatically transcribed into a textual file through speech recognition techniques, and the textual file can be processed as described herein to refine an ontology.
Once one or more initial ontologies 110 are selected and a training data set 205 is developed, the training phase 1 continues by executing a training module 300, example of which is depicted in
Starting with the embodiment of
After the set of significant phrases and significant phrase pairs are obtained whether through method 400 or manually, the significant phrases are then added as ontology terms to the initial ontology 110 and associated with ontology entities at step 302. As used herein, a phrase ϕ=w1, . . . , wn comprises a sequence for words. It can be said that a term T′ that comprises the word sequence w′1, . . . , w′k is contained in ϕ, and denoted by T′⊂ϕ, if k<n and there exists some index i such that: wi=w′1, . . . , wi+k−1=w′k. A pair of terms T′,T″ are mutually contained in ϕ, if both are contained in ϕ (namely T′,T″⊂ϕ) with no overlap between them, namely T′∩T″=Ø.
In certain exemplary embodiments, given a set of phrases Φ, training module 300 performs step 302 by sorting the phrases according to their length (shorter phrases are processed first), and then for each ϕ∈Φ, and performing the following:
Following step 302, at step 304 new abstract relations are then added to the initial ontology 110 using the obtained set of significant phrase pairs. As used herein, the set of significant phrase pairs are denoted Ψ⊆Φ×Φ, where ω:Ψ→+ is a scoring function that associates a score to a phrase pair.
In certain exemplary embodiments, training module 300 performs step 304 in the following way:
After adding the new abstract relations to the ontology at step 304, training module 300 then adds new relation instances to the ontology based on the significant phrase pairs Ψ and the scoring function ω:Ψ→+ at step 306. Training module 300 performs step 306 in, for example, the following way: Iterate over all pairs (ϕ1,ϕ2)∈Ψ. If there exists T1,T2∈ that correspond to ϕ1,ϕ2, respectively, compute an entity pair E1*,E2* such that
Namely, select the most specific abstract relation E1*,E2* that corresponds to the term pair T1,T2.
Upon completion of the adding of new relation instances at step 306, the training phase 1 may be completed 308 and the training module 300 may output and store the refined ontological structure at step 310, which is a refined version of the initial ontology 110 referred to in the discussion of
Exemplary method 400 beings by accepting as inputs a generic language model LG and a set of documents, wherein generic model LG is a model that is supposed to model the language distribution of generic texts that are not specific to the common source or its associated field of interest.
For ease of description and conception, the exemplary process 400 is divided into four exemplary phases 402, 404, 406, and 408, three of which can be used for the extraction of significant phrases 402, 404, and 406, and one of which can be used in for the extraction of significant phrase co-occurrences. However, such divisions are not intended to be a limiting of the embodiment or the invention. In certain embodiments not all of the phases need be performed.
In regards to the extraction of a set of significant phrases, an exemplary method 400 can include, for example: first, exemplary language-model generation phase (step 402); second, exemplary candidate accumulation phase (step 404), and third, exemplary significant phrase filtering phase (step 406).
As used, L(w1, . . . , wm) denotes the log-probability of the word sequence w1, . . . , wm as induced by the language model L. For example, if L is a trigram model this log-probability can be expressed as:
The language-model generation phase (step 402) can include, for example, iterating over the input documents and subdividing each document into meaning units. Meaning units are sequences of words that express an idea. In the context of spoken or informal communications, the meaning unit may be the equivalent of a sentence. Meaning units can divide scripts or utterances into a basic segments of meaning or the equivalent of a sentence, when narrated text is compared to written text. A meaning unit may be a sequence of words spoken by one speaker in a conversation without interference. A non-limiting example of a meaning unit in a customer service context would be the customer statement “I would like to buy a phone.” In some embodiments, the meaning unit may include some level of speaker interference, e.g. very short acknowledgement statements by the other speaker. All terms in the meaning unit are linked within the boundaries of the meaning unit. In certain embodiments the subdividing above is induced by punctuation marks. However, if the input texts are generated by transcribing audio data, the subdividing can be performed using a zoning algorithm; see for example the zoning algorithm described in U.S. patent application Ser. No. 14/467,783.
Once the meaning units have been subdivided, the Language-model generation phase (step 402) can process each of the meaning units and count the number of n-grams up to some predetermined order (unigrams, bigrams, trigrams, etc.). An order of 3 or 4 has been seen to yield good results. Once the number of n-grams are counted, language-model probabilities can be estimated based on the given counters, and a source-specific language model Ls can be obtained. One suitable way of obtaining source-specific language model Ls, is by a applying a suitable smoothing technique; see, for example: S. F. Chen and J. Goodman, An empirical study of smoothing techniques for language modeling, in Computer Speech and Language (1999) volume 13, pages 359-394.
In an exemplary candidate accumulation phase (step 404), a set of candidates C are created, where each candidate is an n-gram (a sequence of n words), and its respective number of occurrences stored. For example, once that the input text documents have been subdivided into meaning units, the following can be performed for each meaning unit:
For each 1≤n≤nmax (nmax is the maximal number of words per phrase):
Once the candidate accumulation phase (step 404) is completed, the significant phrase filtering phase (step 406) can calculate a phase score for each candidate phrase and then keep only those phrases whose score is above a threshold. The candidate accumulation phase (step 404) can be performed by, for example, iterating over the n-grams in , let f(w1, . . . ,wn) be the frequency of the phrase w1, . . . , wn, which can be computed by the counter stored at normalized by the total number of words encountered in all text documents. The overall phrase score for the candidate w1, . . . , wn can computed in, for example, the following manner:
Φ(w1, . . . ,wn)=αP·P(w1, . . . ,wn)+αS·S(w1, . . . ,wn)+αf·log f(w1, . . . ,wn)
The significant phrase filtering phase 406 only keeps those phrases for which Φ(w1, . . . , wn)>τΦ. Where τΦ is a threshold for the overall phrase score αP, αS and αf are scaling parameters that can be used to give more significance to one of the measures over another, these may be optional.
After the set of significant terms are extracted, method 400 can continue to the fourth phase 408 where significant phrase co-occurrences are extracted. Namely, those pairs of phrases that tend to co-occur in the same meaning unit, possible up to some distance. The extraction of significant phrase co-occurrences can begin by, for example, iterating over all meaning units and locating the occurrences of the individual phrases. In certain embodiments longer phrases are preferred over shorter ones. In case of overlapping occurrences of a pair of phrases, only the occurrence of the longer phrases is kept. As used herein, c(ϕ) denotes the number of occurrences of the phrase ϕ. The number of co-occurrence of pairs of phrases in the same meaning unit is counted. Depending on parameter, one may count only pairs of phrases that are separated by m words at most. As used herein, c(ϕ1,ϕ2) denotes the number of co-occurrences of the phrases ϕ1 and ϕ2. Depending on another parameter, this counter may or may not be sensitive to order.
Based on the counter values, it is possible to compute the probability p(ϕ) of a phrase and the probability p(ϕ1,ϕ2) of the co-occurrence of a pair of phrases. The log-likelihoods of the co-occurrence of phrases ϕ1 and ϕ2 can be defined as follows:
After calculating the log-likelihoods, phase 408 then identifies a co-occurrence of a pair of phases ϕ1,ϕ2 if l(ϕ1,ϕ2)>τl where τl is a log-likelihood threshold.
In summary, the exemplary process 400 described above is capable of extracting a set of significant phrases and a set of significant phrase co-occurrences from an input set of text documents that related to some specific domain. Since phrases must be significantly more prominent in the processed texts with respect to the generic model, the prominence score can help filter very frequency phrases in the language that carry no special significance in the specific domain. Similarly, the stickiness score can help filter false phrases that are induced by an incidental concatenation of a pair of shorter terms. Process 400 can be used in in ontology refining process described in
Referring back to
More specifically, once the initial ontology 110 has been refined in training stage 1, the system can use a tagging process 2 to tag meaningful elements in incoming communications data, such as transcribed interactions, and then store the tagged data for use by a analytics module 3. In certain embodiments for instance, the tagging process 2 can include loading key ontology data into the tagging system, and then executing the process depicted at
Once the refined ontology data has been loaded into the tagger's internal memory, the fetch/tag/store process begins. In the exemplary embodiment of
In one embodiment, for each data set (or bulk of datasets, depending on the system configuration) the system tags specific elements and then saves the interaction in a temporary repository, the Context module 321. The process tagging process can be repeated for each element—scripts 324, zoning 326 (meaning unit tagging), and relations 328. After the interactions have been tagged with relations 328, the Write to Database module 333 processes the tagged interaction and sends it to the database for storing 330. As described above, the data set can include any number of types and formats of data. For example, the data set may include audio recordings, transcribed audio, email communications, text message communications, etc.
The tagged communications data can then be used to generate any number of analytics 3 (see
Any variety of data can be processed and tagged.
Although the computing system 1200 as depicted in
The processing system 1206 can comprise a microprocessor and other circuitry that retrieves and executes software 1202 from storage system 1204. Processing system 1206 can be implemented within a single processing device but can also be distributed across multiple processing devices or sub-systems that cooperate in existing program instructions. Examples of processing system 1206 include general purpose central processing units, applications specific processors, and logic devices, as well as any other type of processing device, combinations of processing devices, or variations thereof.
The storage system 1204 can comprise any storage media readable by processing system 1206, and capable of storing software 1202. The storage system 1204 can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage system 1204 can be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems. Storage system 1204 can further include additional elements, such a controller capable, of communicating with the processing system 1206.
Examples of storage media include random access memory, read only memory, magnetic discs, optical discs, flash memory, virtual memory, and non-virtual memory, magnetic sets, magnetic tape, magnetic disc storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that may be accessed by an instruction execution system, as well as any combination or variation thereof, or any other type of storage medium. In some implementations, the storage media can be a non-transitory storage media. In some implementations, at least a portion of the storage media may be transitory. It should be understood that in no case is the storage media merely a propagated signal.
User interface 1210 can include a mouse, a keyboard, a voice input device, a touch input device for receiving a gesture from a user, a motion input device for detecting non-touch gestures and other motions by a user, and other comparable input devices and associated processing elements capable of receiving user input from a user. Output devices such as a video display or graphical display can display an interface further associated with embodiments of the system and method as disclosed herein. Speakers, printers, haptic devices and other types of output devices may also be included in the user interface 1210.
As described in further detail herein, the computing system 1200 receives communication data 10. The communication data 10 may be, for example, an audio recording or a conversation, which may exemplarily be between two speakers, although the audio recording may be any of a variety of other audio records, including multiple speakers, a single speaker, or an automated or recorded auditory message. The audio file may exemplarily be a .WAV file, but may also be other types of audio files, exemplarily in a pulse code modulated (PCM) format and an example may include linear pulse code modulated (LPCM) audio data. Furthermore, the audio data is exemplarily mono audio data; however, it is recognized that embodiments of the method as disclosed herein may also be used with stereo audio data. In still further embodiments, the communication data 10 may be streaming audio or video data received in real time or near-real time by the computing system 1200.
This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.
The present application is based on and claims priority to U.S. Provisional Patent Application Ser. No. 62/108,264, filed Jan. 27, 2015 and U.S. Provisional Patent Application Ser. No. 62/108,229, filed Jan. 27, 2015, the disclosures of which are incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5317673 | Cohen et al. | May 1994 | A |
5737617 | Bernth et al. | Apr 1998 | A |
6076088 | Paik et al. | Jun 2000 | A |
6434557 | Egilsson et al. | Aug 2002 | B1 |
6560590 | Shwe et al. | May 2003 | B1 |
6600821 | Chan et al. | Jul 2003 | B1 |
6718296 | Reynolds et al. | Apr 2004 | B1 |
7113958 | Lantrip et al. | Sep 2006 | B1 |
7552053 | Gao et al. | Jun 2009 | B2 |
7734467 | Gao et al. | Jun 2010 | B2 |
7844459 | Budde et al. | Nov 2010 | B2 |
7853544 | Scott et al. | Dec 2010 | B2 |
7904414 | Isaacs | Mar 2011 | B2 |
7912701 | Gray et al. | Mar 2011 | B1 |
8036876 | Sanfilippo et al. | Oct 2011 | B2 |
8078565 | Arseneault et al. | Dec 2011 | B2 |
8160232 | Isaacs et al. | Apr 2012 | B2 |
8190628 | Yang et al. | May 2012 | B1 |
8260809 | Platt et al. | Sep 2012 | B2 |
8285552 | Wang et al. | Oct 2012 | B2 |
8447604 | Chang | May 2013 | B1 |
8825488 | Scoggins et al. | Sep 2014 | B2 |
8825489 | Scoggins et al. | Sep 2014 | B2 |
8874432 | Qi et al. | Oct 2014 | B2 |
9066049 | Scoggins et al. | Jun 2015 | B2 |
9189525 | Acharya et al. | Nov 2015 | B2 |
9232063 | Romano et al. | Jan 2016 | B2 |
9355354 | Isaacs et al. | May 2016 | B2 |
9477752 | Romano | Oct 2016 | B1 |
9569743 | Fehr et al. | Feb 2017 | B2 |
9575936 | Romano et al. | Feb 2017 | B2 |
9639520 | Yishay | May 2017 | B2 |
9646605 | Biatov et al. | May 2017 | B2 |
9697246 | Romano et al. | Jul 2017 | B1 |
9720907 | Bangalore et al. | Aug 2017 | B2 |
9760546 | Galle | Sep 2017 | B2 |
20020032564 | Ehsani | Mar 2002 | A1 |
20020128821 | Ehsani et al. | Sep 2002 | A1 |
20020188599 | McGreevy | Dec 2002 | A1 |
20030028512 | Stensmo | Feb 2003 | A1 |
20030126561 | Woehler et al. | Jul 2003 | A1 |
20040078190 | Fass | Apr 2004 | A1 |
20040236737 | Weissman | Nov 2004 | A1 |
20060248049 | Cao et al. | Nov 2006 | A1 |
20070016863 | Qu et al. | Jan 2007 | A1 |
20070118357 | Kasravi | May 2007 | A1 |
20080021700 | Moitra et al. | Jan 2008 | A1 |
20080154578 | Xu et al. | Jun 2008 | A1 |
20090012842 | Srinivasan | Jan 2009 | A1 |
20090043581 | Abbott et al. | Feb 2009 | A1 |
20090099996 | Stefik | Apr 2009 | A1 |
20090150139 | Jianfeng et al. | Jun 2009 | A1 |
20090204609 | Labrou et al. | Aug 2009 | A1 |
20090254877 | Kuriakose et al. | Oct 2009 | A1 |
20090306963 | Prompt et al. | Dec 2009 | A1 |
20090326947 | Arnold et al. | Dec 2009 | A1 |
20100030552 | Chen | Feb 2010 | A1 |
20100057688 | Anovick et al. | Mar 2010 | A1 |
20100161604 | Mintz et al. | Jun 2010 | A1 |
20100275179 | Mengusoglu et al. | Oct 2010 | A1 |
20110161368 | Ishikawa et al. | Jun 2011 | A1 |
20110196670 | Dang et al. | Aug 2011 | A1 |
20120303356 | Boyle et al. | Nov 2012 | A1 |
20130018650 | Moore et al. | Jan 2013 | A1 |
20130066921 | Mark | Mar 2013 | A1 |
20130132442 | Tsatsou | May 2013 | A1 |
20130166303 | Chang et al. | Jun 2013 | A1 |
20130260358 | Lorge et al. | Oct 2013 | A1 |
20140040275 | Dang | Feb 2014 | A1 |
20140075004 | Van Dusen et al. | Mar 2014 | A1 |
20140143157 | Jeffs et al. | May 2014 | A1 |
20140222419 | Romano | Aug 2014 | A1 |
20140297266 | Nielson | Oct 2014 | A1 |
20150066506 | Romano et al. | Mar 2015 | A1 |
20150074124 | Sexton | Mar 2015 | A1 |
20150127652 | Romano | May 2015 | A1 |
20150169746 | Hatami-Hanza | Jun 2015 | A1 |
20150170040 | Berdugo | Jun 2015 | A1 |
20150193532 | Romano | Jul 2015 | A1 |
20150220618 | Horesh | Aug 2015 | A1 |
20150220626 | Carmi et al. | Aug 2015 | A1 |
20150220630 | Romano et al. | Aug 2015 | A1 |
20150220946 | Horesh et al. | Aug 2015 | A1 |
20160055848 | Meruva et al. | Feb 2016 | A1 |
20160078016 | Ng Tari | Mar 2016 | A1 |
20160078860 | Paulik et al. | Mar 2016 | A1 |
20160180437 | Boston | Jun 2016 | A1 |
20160217127 | Segal et al. | Jul 2016 | A1 |
Entry |
---|
Coursey et al, “Topic identification using Wikipedia graph centrality.”, 2009, In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers. Association for Computational Linguistics, 2009. pp. 117-120. |
Kumar et al, “Automatic keyphrase extraction from scientific documents using N-gram filtration technique.”, 2008, Proceedings of the eighth ACM symposium on Document engineering. ACM, 2008., pp. 199-208. |
Chen, S.F., et al, “An Empirical Study of Smoothing Techniques for Language Modeling,” Computer Speech and Language, vol. 13, 1998, 62 pages. |
Král, Pavel, et al., “Dialogue Act Recognition Approaches,” Computing and Informatics, vol. 29, 2010, pp. 227-250. |
Stolcke, Andreas, et al., “Automatic Linguistic Segmentation of Conversational Speech,” IEEE, vol. 2, 1996, pp. 1005-1008. |
Zimmerman, Matthias, et al., “Joint Segmentation and Classification of Dialog Acts in Multiparty Meetings,” Acoustics, Speech and Signal Processing, 2006, pp. 581-584. |
Extended European Search Report, dated Feb. 18, 2015, received in connection with corresponding European Application No. 14182714.7. |
Federico, M., et al., “IRSTLM: an Open Source Toolkit for Handling Large Scale Language Models,” Ninth Annual Conference of the International Speech Communication Association, 2008, pp. 1618-1621. |
Mikolov, T., et al., “Efficient Estimation of Word Representations in Vector Space,” Proceedings of the International Conference on Learning Representations (ICLR), arXiv:1301.3781v3, 2013, 12 pages. |
Ponte, J.M., et al., “Text Segmentation by Topic,” Computer Science Department, University of Massachusetts, Amherst, 1997, 13 pages. |
Ramos, J., “Using TF-IDF to Determine Word Relevance in Document Queries,” Proceedings of the First Instructional Conference on Machine Learning, 2003, 4 pages. |
Rosenfeld, R. “The CMU Statistical Language Modeling Toolkit and its use in the 1994 ARPA CSR Evaluation,” Proceedings of the Spoken Language Systems Technology Workshop, 1995, pp. 47-50. |
Stolcke, A., “SRILM—An Extensible Language Modeling Toolkit,” Seventh International Conference on Spoken Language Processing, 2002, 4 pages. |
Number | Date | Country | |
---|---|---|---|
20160217128 A1 | Jul 2016 | US |
Number | Date | Country | |
---|---|---|---|
62108264 | Jan 2015 | US | |
62108229 | Jan 2015 | US |