Patent Application
20240070434
The present invention relates to using natural language processing (NLP) techniques, large language models (LLMs), transformer-based, deep-learning models, and other techniques to train, analyze, extract and organize domain-specific information from a conventional knowledge base and other sources. Based on the domain-specific information, the present invention further provides an interactive system that responds user queries and that provides domain-specific services,
In the detailed description herein, a “knowledge base” refers to an information system that provides domain-specific information. The knowledge base is typically constituted by a collection of documents, or other forms of information repository on any medium (e.g., audio and video files), that include information specific to a designated domain. The domain may be, for example, a particular business or a particular group of businesses (e.g., businesses within a specific industry). Generally, examples of documents supporting a knowledge base may include one or more websites, email messages, PDF files, or any combination of such documents.
Documents in a knowledge base may be developed for an audience external to the domain, an audience within the domain, or both. For example, a retail company may maintain an order management system that is configured to access both a customer-facing knowledge base and an internal knowledge base that is accessible only by persons—mostly employees—specifically authorized by the retail company. Both knowledge bases may include, for example, the retail company's refund policies.
In many industries, customer service consists primarily of furnishing basic information of a business to its customers. In other words, the role of customer service of a business is to fill gaps in the customers' knowledge concerning the business, especially gaps that frustrate customers from achieving their respective goals favorable to the business. For example:
In the prior art, when a customer seeks any information, the customer is often directed to a specific portion of a knowledge base to find the information sought. That portion of the knowledge base may include, for example, a large document (e.g., a policy document) that requires significant time and effort to understand and, hence, entails an arduous process. As a result, rather than going through the arduous process, the customer often in the first instance resorts to a human customer service executive of the company to have his or her questions answered. The cost of providing such customer service is high. Even when the customer goes through the arduous process, customer satisfaction may be adversely impacted, especially when they fail to find the information sought in the specified portion of the knowledge base.
Thus, in the prior art, knowledge bases primarily merely allow a user to locate documents that potentially contain information sought by the user. However, as a customer typically seeks very specific information, the knowledge base fails the customer by leaving it to the customer to extract the information sought himself from an ordered list of documents returned. For example, when a user asks the question “Who has won the maximum individual medals in Olympics 2012?”— which desirably should elicit the response “Michael Phelps”—the user is instead typically presented with a list of relevant documents for he or she to explore. Naturally, such a process results in an unsatisfactory customer experience, and may even lead to frustration, when the answer sought is not found in the documents presented even after an arduous process.
Likewise, even though Question-Answering (QA) systems have evolved over many years, they still typically rely on rules, keywords, synonyms or pattern matching-based techniques to respond to a query Such methods limit the QA system to responding only to a limited set of anticipated questions. The results are often in low recall with at best average precision. Even though generative models based on artificial intelligence (e.g., GPT-4) have recently been applied to synthesize questions, the quality of the answers from a conventional QA system remains low.
Current knowledge base research is focused predominantly on creating open domain systems (i.e., generic systems that require customization to make the knowledge domain-specific). However, customer service systems are required by necessity to answer domain-specific questions. This disparity makes incorporating solutions from recent knowledge base research a challenge. Few knowledge base systems can be effective enough to serve a specific domain when constituted by a full spectrum of structured, semi-structured and unstructured open-domain information. As a result, customer service systems typically use siloed, unsalable solutions that require high maintenance at high operating costs.
According to one embodiment of the present invention, an information system (“conversational knowledge base) capable of responding to user query incorporates contemporaneous advancements in NLP, LLMs and deep learning (e.g., transformer-based deep-learning models) to extract information from its documents. The conversational knowledge base significantly enhances end user experience by concisely presenting relevant information accurately and practically instantly. In one embodiment, a parser in the conversational knowledge base parses the documents from various sources to produce, substantially without human intervention, precise answers to synthesized questions using transformer-based deep learning models. Thus, a customer may discover relevant information without being required to navigate a large knowledge base themselves, thereby significantly improving customer experience.
In addition, a conversational knowledge base of the present invention is easier to train than a customer service system based on conventional topic-based algorithms. In one embodiment, owing to its extractive algorithms and heuristics, together with its custom-built answer-matching algorithms, a conversational knowledge base of the present invention generates higher quality and more accurate answers. Thus, a conversational knowledge base of the present invention provides a pleasant user experience from the perspectives of end-users (e.g., customers) and administrators alike. The end-user experience is enhanced by the system's prompt and precise responses, which are relevant information presented in compatible formats. At the same time, from the perspective of the administrator, a conversational knowledge base of the present invention may be set up quickly and provides transparency and smooth operational control. Many algorithms in a conversational knowledge base of the present invention are based on machine learning and artificial intelligence.
The present invention is better understood upon consideration of the detailed description below in conjunction with the accompanying drawings.
According to one embodiment of the present invention, a conversational knowledge base may incorporate any domain-specific business information from any source business documents (e.g., frequently asked questions (“FAQs”), how-to guides, and trouble-shooting instructions). The conversational knowledge base may also allow a user to search for proposed solutions to problems likely encountered by participants in the domain, while alleviating the burden of reviewing a large number of documents by the user. Using algorithms of artificial intelligence (AI) and Human-AI interactions, the conversational knowledge base is configured to achieve, for example, the following features and goals:
As shown in
A conversational knowledge base of the present invention may offer a graphical user interface (GUI) to facilitate an administrator to gather documents at step 101. In customer service system 500 of
Through application program 509 and an API of data access layer 508, an administrator can also manage the conversational knowledge base in customer service system 500. For example, at step 102 of
The conversational knowledge base may utilize one or more data-extraction techniques (e.g., crawlers, scrapers and rich document readers) to extract structured and unstructured information from various data sources, as mentioned above. Open-source and other libraries (e.g., Selenium, Beautiful Soup, and Textract) may be used to extract data from different sources, including text sources. For uniformity and consistency, the parsed information is expressed substantially into four key types: title (i.e., subject area of interest or “key topic”), content, URL and meta information. In this regard, “content” refers to key text information. The other parsed information types are optional, unless they themselves represent content, as when extracted from certain sources. The conversational knowledge base parser may extract, merge and store the parsed information from any number or kind of sources. Each source may be periodically automatically refreshed at short intervals, so as to eliminate or reduce any manual intervention required to update the information of the conversational knowledge base. Additionally, the administrator also may define custom parsers to retrieve the required information. Any or all of these techniques and operations may be implemented in customer service system 500 of
At steps 103a and 103b, the conversational knowledge base articles synthesize a diverse and exhaustive set of question-answer pairs. For example, based on the extracted information, question-answer pairs are generated in customer system 500 of
According to one embodiment of the present invention, a question generation model—based on a custom or an open-source algorithm—generates a diverse and exhaustive set of close-ended or open-ended questions. For example, in one embodiment, the conversational knowledge base uses a transformer-based deep learning pre-trained-t5-small model, which is fine-tuned on the popular Squad dataset for end-to-end question generation. The transformer-based deep learning model may also be further fine-tuned on client-specific datasets for improved results. Fine-tuning may include extracting a list of consecutive pairs of sentences from the text and passing the extracted sentences into the model for use as context for the questions generated. Relevant context may also be extracted by tokenizing the available text. The set of generated questions may be refined or pruned, as desired, using a rule-based approach, for example, to exclude questions with low confidence scores, or to enhance the questions to a more desirable manner of speech (e.g., active or passive voice) to work with.
For a given collection of conversational knowledge base articles and questions extracted from sources in the conversational knowledge base, both short and long answers may be generated to each question using state-of-the-art deep learning-based answer retrieval algorithms. In one embodiment, short answers are first generated using maximum similarity (e.g., cosine, entailment measure) from text that is split into different relevant sections (e.g., divided into consecutive sentence sections, or into tokens or into characters). Each consecutive sentence section may be provided as context and as a potential long answer. The boundaries of a potential long answer may be refined using custom rule-based methods, as mentioned above. The machine representation (“embeddings”) used for selecting a relevant short answer may be expressed as a vector or as a transformer-based or custom embedding. For example, the conversation knowledge base may use embeddings based on multiQA-cosv1 to retrieve the most relevant short answer.
By automating the tasks of question generation and their answers, an exhaustive set of questions may be generated, which is virtually impossible if questions were manually added to the knowledge base one-by-one.
An administrator of a conversational knowledge base may be provided with an option to review and refine custom entities generated from a corpus of the knowledge using traditional and custom named entity recognition (NER) algorithms. In some embodiments, predefined global entities may be provided for operational ease. Such custom and global entities may be extracted from the question-answer pairs generated using a custom-built algorithm. The custom-built algorithm may extract the entities based on a context present in the utterance, for example. An administrator can then create customized response templates using the entity types as placeholders. The templates may be used by the conversational knowledge base to respond to customer queries by populating user-specific data into the entity placeholders at run-time.
AI may be used to generate various questions and answers. Operational controls are provided to review and approve generated questions and answers by the administrator. These pre-generated questions answers are used to respond to customer queries at the run time. Again, AI is used to find the closest answer to the customer query by looking at pre-generated questions and answers. Additionally, pre-generated answers can be short form, as well as long form. Depending on the conversation channel used, answers provided may vary. For example, short form answers may be used for a chat widget to meet the requirements for the user experience. Similarly, long form answers may be used if conversation takes place via email where long form text may be acceptable.
Question-answer pairs created are mapped into different topics of interest, so that similar question-answer pairs may be grouped into a cluster under the same topic and independently accessible apart from question-answer pairs of other topics. Clustering may be achieved using, for example, a clustering algorithm that leverages a discriminative clustering model. Super-clusters and sub-clusters, as understood by those of ordinary skill in the art, may also be created. Other algorithms (e.g., centroid-based, density-based, distribution-based, entity-based, and hierarchical clustering algorithms) may also be used for creating clusters of question-answer pairs. The topics are identified based on the distribution of keywords in each cluster. The set of question-answer pairs and corresponding topics are sent to the moderator for quality review, so that customers may receive high-quality and moderated answers. Furthermore, question-answer clustering allows the administrator to prioritize review and training based on the volume of clusters.
In summary, therefore, AI-digested knowledge base 514 in customer service system 500 of
At step 105, the question-answer pairs are accessed for review, for supervised and unsupervised training, and for moderation by one or more administrators to ensure high-quality. In customer service system 500 of
Conversational knowledge base 106 provides the bases for responding to customer queries (steps 107-110). Customers typically access a conversational knowledge base of the present invention through an application program that is integrated to the conversational knowledge base through, for example, an API. In customer service system 500 of
When a customer query is received at step 107, NLP processor 515, using NLP techniques, calls upon user intent classification module 516 to determine the user's intent (e.g., to ask about tracking a delivery). Once the user's intent is ascertained, question-answer prediction module 517, operating various knowledge base search algorithms, identifies and retrieves from AI-digested knowledge base 514 candidate question-answer pairs suitable for responding to the customer's query. As illustrated in
According to one embodiment, a conversational knowledge base of the present invention responds to customer queries using an ensemble approach that ranks and outputs the best-matched question-answer pairs based on assessing: (i) query-answer semantic similarity and (ii) query-question semantic similarity to the synthesized question-answer pairs in the conversational knowledge base. For example, a best predetermined number of question-answer pairs are first obtained on the query-answer semantic similarity basis using, for example, a keyword-based search on the customer query—after applying on the customer query suitable pruning techniques (e.g., lower-casing, lemmatization, stemming and removal of stop words)—and ranking by a relevancy score. The conversational knowledge base then selects from the best predetermined number of question-answer pairs the answer that is most relevant to the customer query and that has a relevancy score exceeding a predetermined threshold. If no question can be selected on the query-answer semantic similarity basis, the same selection process is applied to the best question-answer pairs based on query-question similarity basis. If a relevant answer is still not found after selection using both semantic similarity bases, a predetermined number of relevant conversational knowledge base articles may be suggested to the customer to review, so as to avoid a human-handoff. At the customer's request, the customer query may be referred to a human agent, to ensure the customer receives a satisfactory resolution.
The agent may participate through any customer service platform that has access to the conversational knowledge base and that supports simultaneous interaction with the customer. For example, as illustrated in
As shown also in
As shown in
One of the retrieved question-answer pairs may either (i) be provided directly as a response to the customer, or (ii) upon recognizing the intent of the customer based on the customer query, channel the interaction with the customer into a customized workflow (step 109). The response to the customer may be in a short format (e.g., if the customer query is posed to a live interactive chatbot), or in a long format (e.g., if the customer query is posed in an email message, or any non-interactive format). The long format allows the response to be given in greater detail, for example, with cross-reference links to other relevant topics.
In one embodiment, as shown in
Besides answering the customer's query, the customized workflow may provide additional services at step 110 relevant to the intent of the customer. For example, if the customer query concerns when a refund would be paid after returning a product, the customized workflow would also take the customer into a sequence of steps to complete the product return process (e.g., taking the customer step-by-step from retrieving the purchase order up to and including printing a shipping label for returning the product by courier). As shown in
In one embodiment, the accuracy of the knowledge base search algorithms in assessing customer intent and the reception of the resulting response delivered to the customer may be fed back to review, moderation and training steps in step 105. As in the document gathering at step 101, the conversational knowledge base may offer a GUI for the administrator to efficiently moderate and monitor the positive impact on the end customers. Such feedback may be facilitated by a report generated in the conversational knowledge base. The report may include tracking metrics such as customer query or question posed, topic of interest, URL visited, question posed, long-form or short-form answer delivered, the source in the conversational knowledge base utilized and its identification, number of answers, articles or links displayed to the customer, number of answers, articles or links accepted (e.g., links followed) by the customer, ratings of the answers by the customer, feedback comment made by the customer, number of question-answer pairs reviewed, modified or deleted by an administrator. The administrator may be able to view, download or share the analytics report to help track and take necessary action to enhance the performance of the conversational knowledge. To facilitate the administrator, information in the analytics report may be selected using custom filters, based on variables such as time range, and data collection environment (e.g., live or test/sandbox utilization). Question-answer pairs based on customer reviews are then released by the administrator for use in future responses to customers.
In one embodiment, all answers to customer queries generated by the conversational knowledge base are sent to an administrator to review, along with the customer query, the identity of the customer, if known, and the circumstances under which the customer question is posed. The administrator may incorporate a reviewed answer for training and approval in the conversational knowledge base. Prior to incorporation, the administrator may edit both the incorporated customer query and the provided answer. In addition to enhancing the question-answer bank or library, the feedback loop allows machine learning to improve performance and the metrics by which the proposed answers can be ranked.
The administrator may also select question-answer pairs from the conversational knowledge base to perform a manual (i.e., under human supervision) or semi-supervised (i.e., AI-assisted) curation process. The curation process ensures that the curated question-answer pairs comply with policy and compliance requirements.
The administrator may also create templates into which answers may be embedded. These templates allow customization for use with different suitable situations at run-time. For example, in a situation where the long-form answer may be appropriate, and the template may be instantiated for that situation in the appropriate format. Likewise, a template may be instantiated for different business entities (e.g., different products) or for different customers. Templates are particularly useful for creation of generated answer multi-step, personalized workflow responses, that can be subsequently instantiated with various desired degrees of customization (e.g., additional greetings and follow-up steps, in addition to providing the answer) under different situations. Templates are also flexible when one or more feedback loops at different points of in workflow are required to accommodate different actions that can be taken (e.g., presenting additional options, answers, greater details), incorporating the contexts at those points in the workflow.
The administrator is provided an interface to an optimizer that allows the administrator to inspect high-volume or high-value queries, queries associated with specific topics, keywords or entities, or queries that conform to specific volume, usage or feedback profiles. Such queries are surfaced by the reporting, or other in-platform search or discovery tools. One example of a high-value query is a query that invokes one or more specific entities or keywords of interest, or a query that elicits a specific response, or a response that includes a specific resource (e.g., by identified by a specific URL, a specific webpage or document).
The optimizer is particularly useful when certain queries become high-volume unexpectedly. The optimizer also provides user-feedback that may prompt the administrator to offer the users different responses to the same query in a workflow, according to their preferences. Upon surfacing these queries and answers, the administrator may be provided different options to perform additional customization of the response. The optimizer identifies for the administrator the responses on which the administrator's efforts are best spent.
In one embodiment, the administrator may:
The conversational knowledge base of the present invention may incorporate a multi-lingual service to handle knowledge bases of multiple languages. The multi-lingual service may obtain translation of non-native language documents and index the translated native language versions (e.g., English). The translated versions may facilitate use by other services (e.g., other AI services) in, for example, generations of questions and answers.
When a non-native language document is encountered, the multi-lingual service may involve one or more translations of the documents to create a version of the document of a preferred language. This translation process ensures that the information within the document can be effectively processed and utilized for further processing.
At run time, the multi-lingual service may convert a non-English user question into English for processing. The response to the converted English question may be in English. The multi-lingual service may translate the English response to the user's language in which the question is posed.
The above detailed description is provided to illustrate specific embodiments of the present invention and is not to be taken as limiting. Numerous variations and modifications within the scope of the present invention are possible. The present invention is set forth in the following accompanying claims.
The present application is related to and claims priority of U.S. provisional application (“Provisional Application”), Ser. No. 63/402,859, entitled “CONVERSATIONAL KNOWLEDGE BASE,” filed on Aug. 31, 2022. The disclosure of the Provisional Application is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63402859 | Aug 2022 | US |
