Patent Application
20240028927
The field relates generally to information processing systems, and more particularly to feedback data analysis in such information processing systems.
Many information processing systems associated with enterprises are configured to receive feedback data from entities (e.g., users) for the purpose of adapting the information processing systems and/or one or more other aspects of the enterprises. However, oftentimes, the feedback data is misdirected within an enterprise and/or misinterpreted by the enterprise. As a result, from the perspective of users who do not detect any adaptation in the information processing system and/or the enterprise itself, there is a feeling that the feedback data has been essentially ignored. This can have a significant negative impact on the information processing system and/or the enterprise.
Embodiments provide techniques for domain-driven feedback data analysis.
For example, according to one illustrative embodiment, a method comprises obtaining a feedback data set, and classifying the feedback data set into at least one domain of a plurality of domains. The feedback data set is mapped to a domain data set corresponding to the at least one domain, and a root cause is computed for the feedback data set to enable one or more actions to be taken in response to the feedback data set based on the root cause.
In some illustrative embodiments, classifying the feedback data set into at least one domain of the plurality of domains may further comprise computing at least one intent attribute and at least one sentiment attribute from the feedback data set. Then, computing the root cause for the feedback data set may further comprise utilizing the at least one intent attribute, the at least one sentiment attribute, and the domain data set to compute a decision on the root cause for the feedback data set. In some illustrative embodiments, the method may further comprise causing the one or more actions to be taken in response to the feedback data set based on the root cause.
Further illustrative embodiments are provided in the form of a non-transitory computer-readable storage medium having embodied therein executable program code that when executed by a processor causes the processor to perform the above steps. Additional illustrative embodiments comprise an apparatus with a processor and a memory configured to perform the above steps.
Advantageously, illustrative embodiments obtain the feedback data set from a customer engaged with an enterprise through one or more transactions, and the plurality of classification domains comprise a plurality of supply-chain domains such that the computed root cause is used to enable one or more actions to be taken in a supply chain managed by the enterprise to address one of a customer intent and a customer sentiment extracted from the feedback data set.
These and other features and advantages of embodiments described herein will become more apparent from the accompanying drawings and the following detailed description.
Illustrative embodiments will now be described herein in detail with reference to the accompanying drawings. Although the drawings and accompanying descriptions illustrate some embodiments, it is to be appreciated that alternative embodiments are not to be construed as limited by the embodiments illustrated herein. Furthermore, as used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “based on” is to be read as “based at least in part on.” The term “an embodiment” and “the embodiment” are to be read as “at least one example embodiment.” The terms “first,” “second,” and the like may refer to different or the same objects. Other definitions, either explicit or implicit, may be included below.
As mentioned, many information processing systems associated with enterprises are configured to receive feedback data from entities (e.g., users) for the purpose of adapting the information processing systems and/or one or more other aspects of the enterprises. For example, assuming that the users are customers of a company, such as an original equipment manufacturer or OEM, studies have shown that a majority of people worldwide believe that companies need to take action on the feedback provided by customers. Studies have also shown that nearly all respondents say customer service is essential in their choice of loyalty to a product. Thus, obtaining customer feedback from consumers is paramount for a company or other enterprises. Most companies collect enormous amounts of feedback from their customers, but many do not use it effectively to improve their products and services. It has been shown that poor customer service experiences lead a majority of people to stop doing business with a company. Poor customer service experience is one of the significant outcomes when customer feedback is not correctly analyzed.
The usefulness of feedback is not limited only to providing a better customer experience. It is also a great platform for a company to build a strong relationship with suppliers/vendors (and others in the product supply chain) and recommend to suppliers/vendors to improve their product quality, e.g., in terms of a better way to enhance the procurement experience. Still further, managing feedback can also improve performance of total availability market (TAM) analysis.
Feedback analysis is thus one of the most critical steps once a company has collected customer input, and can provide many benefits including, but not limited to: (i) improving a company's growth; (ii) building an optimal supply chain; (iii) providing insights to suppliers and vendors to improve their quality of products; (iv) improving part and sub-assembly consumption; (v) finding new and better suppliers; and (vi) changing the strategy to allocate rewards among suppliers.
Existing feedback analysis focuses on a general-purpose product rating and customer experience rating. In an organization, the knowledge acquired by specific domain expertise is immersive. For example, in the OEM context, domains such as order management, fulfilment, manufacturing, logistics, quality control, and procurement are the backbone of selling the product. Customer feedback applies to all these domains. Today, these domains are not directly analyzing the input in their domain context. Since customer feedback may be provided using an e-commerce website or other media, the organization's e-commerce team performs silo analysis. The e-commerce team then provides feedback to all domains. Since, for these domains, subject matter experts (SMEs) are not directly involved in the analysis, the feedback is not typically considered, prioritized, and observed within their respective domain context.
Often, organizational decisions depend on the SME. If the feedback analysis is not trained with the proper domain data, the purview of domain SMEs (e.g., data engineers, scientists, etc.) will lead to delayed action and unclear decisions. Thus, general-purpose feedback analysis fails to spot domain-related feedback and misinterprets the feedback since the analysis process is not trained in any domain-specific context.
Illustrative embodiments address the above and other technical issues by providing a domain-driven based feedback data analyzer system and process which, inter alia, intelligently assists, learns, and builds expertise by processing the feedback on the domain context.
Advantageously, in an OEM supply chain context by way of example only, a domain-driven feedback data analysis system according to one or more illustrative embodiments can identify the parts or sub-assembly and corresponding symptoms from feedback and proactively assist the respective domain professional to measure and optimize the components or sub-assemblies, behavior, quality, and price. More particularly, a domain-driven feedback data analysis process according to an illustrative embodiment is configured to apply a context that closely relates to a classified domain, for example: (i) a procurement domain applies parts or sub-assembly of the product performance; (ii) a manufacturing domain involves the parts or sub-assembly of the product behavior; (iii) a pricing domain applies product pricing related feedback; and (iv) a logistics domain uses product shipping related feedback.
Domain context analysis according to one or more illustrative embodiments connects respective domain data repositories and further analyzes and computes the root cause of an issue. Sentiment analysis according to one or more illustrative embodiments, including emotion recognition, can be used to find a part or sub-assembly performance or behavior. Intent analysis according to one or more illustrative embodiments is performed and leveraged from various feedback platforms such as, but not limited to, an e-commerce site, chat, and social media. Multi-modal (e.g., multiple data modalities and even multilingual data) communication, e.g., text to text, text to speech, and speech to text, are supported in one or more illustrative embodiments.
Referring initially to
As further shown, domain-driven feedback data analysis system 100 comprises a feedback communication (comm.) interface 110 which receives the feedback data from system/network 102 and converts the feedback data input from one or more given modalities (e.g., speech, video, image, etc.) into a feedback data in another given modality (e.g., text). The feedback data set is stored in feedback data store 112.
Domain-driven feedback data analysis system 100 further comprises a feedback domain analysis module 120 which, itself, comprises a domain classification module 122 and a domain corpus store 124. Domain classification module 122 and domain corpus store 124 are configured to classify a feedback data set, received from feedback data store 112, into a given domain of a plurality of domains. As will be further explained in detail herein, domain classification may comprise computation of one or more intent parameters and one or more sentiment parameters from the feedback data set. This intent and sentiment parameters enable domain classification module 122 to classify the feedback data set into one or more of the plurality of domains.
Still further, domain-driven feedback data analysis system 100 comprises a domain data analysis module 130 which, itself, comprises a domain mapping module 132, a domain data store 134, a domain diagnosis analysis module 136, and a root cause/action initiation module 138, operatively coupled as shown. Domain mapping module 132 maps the feedback data set to a domain data set from domain data store 134 corresponding to the given domain identified by domain classification module 122. Domain diagnosis analysis module 136 then computes a root cause for the feedback data set to enable one or more actions to be taken, in accordance with root cause/action initiation module 138, in response to the feedback data set based on the root cause. Domain data store 134 represents an intelligent domain data repository for data shared from other domains as well as learned from processing performed by domain-driven feedback data analysis system 100. More particularly, domain data store 134 is a knowledge repository and configured as an expert system learning over time in domain contexts.
Advantageously, unlike existing feedback analysis approaches, domain-driven feedback data analysis system 100 is not general-purpose with only generalized feedback data, but rather it is an intelligent domain-driven and domain-connected system configured and trained for the specific domains in a computing environment in which it is deployed, by way of example only, in a supply chain computing environment. This is achieved by initially configuring each domain with a domain-driven feedback data analysis system 100 and, over a given period of time, each domain-driven feedback data analysis system 100 accumulates expertise associated with every domain through the variety of feedback data received, as well as data shared by other domains.
Also, it is understood that there is collaboration and domain data sharing during the process between domain-driven feedback data analysis system 100 in each of the plurality of domains 202-208. For example, procurement domain 202 may obtain supporting domain information from product domain 204 (e.g., bill or material ort BOM), manufacturing domain 206 (e.g., product build details), and logistics domain 208 (e.g., route/country and other shipping details). Similarly, manufacturing domain 206 may receive supporting domain information from one or more of procurement domain 202, product domain 204, and logistics domain 208, and so on as illustrated. Thus, eventually, each domain-driven feedback data analysis system 100 within a given domain will be trained on domain data from every other domain, and can then respond to any type of feedback data it receives.
In general, domain-driven feedback data analysis reads and understands information and/or other communications from users (e.g., customers via system/network 102 in
“Device is lightweight and fine for light productivity use, school work, etc., but can occasionally slows with heavy to mid-heavy workloads. The CPU is the limiting factor understandably. The operating system is responsive and frequently updated. Screen is very good but some improvement could be made for movies and photo viewing, such as increased contrast and color depth. WiFi stability is decent but could be improved.”
With reference back to
These classification results are then provided to domain data analysis module 130. Domain data analysis module 130 maps the various classifications to domain-specific data (e.g., domain data store 134) for each domain. Thus, continuing with the example customer review, domain data analysis module 130 identifies data in the procurement domain relating to the supplier of the CPU sub-assembly and the supplier of the display. Similarly, for the manufacturing domain, domain data analysis module 130 identifies data relating to CPU contract manufacturer log analysis. Still further, for the product domain, domain data analysis module 130 identifies data relating to laptop and CPU sub-assembly and burn log analysis. Domain data analysis module 130 then uses the domain-specific data to diagnose (e.g., domain diagnosis analysis module 136) one or more issues (root causes) related to the feedback data set, and identifies one or more actions to be taken to remedy the issues in the supply chain. For example, actions can be initiated in the various domains to improve laptop display and/or CPU sub-assemblies, improve early anomaly detection, and increase CPU chip performance. This may involve, by way of example only, domain data analysis module 130 (e.g., root cause/action initiation module 138) generating and transmitting one or more messages across the supply chain computing network to parties and/or systems that can affect the prescribed actions.
Accordingly, illustrative embodiments implement functionalities in domain-driven feedback data analysis system 100 to understand the feedback in a domain context using natural language processing (NLP) and sentiment analysis, and by building a domain expertise repository, i.e., domain-specific data/domain data store 134, for each domain. Feedback communication interface 110 receives the feedback information in one or more data modalities such as in text, video, or audio formats. It is to be appreciated that feedback communication interface 110 is configured to perform audio classification, speech recognition, and/or speech synthesis. Feedback domain analysis module 120 is configured to perform text analysis including entity recognition, intent, sentiment, and content classifications, as well as video processing for facial sentiment analysis and building one or more facial expression recognition models.
An NLP interpreter can be implemented in one or more illustrative embodiments to interpret the language into a specific vocabulary, correct misspellings, implement word synonyms, decode complicated abbreviations, etc., as well as identifying specified restrictions and grammar cloning, rule filtering, searching, grammar pruning, processing, and restriction filtering.
In some illustrative embodiments, audio/video activity detection processing can be implemented to identify and segregate voices from an input data signal. Further, speech signal processing can be performed to detect emotions of the speakers involved in the feedback conversation.
Additionally, in some illustrative embodiments, image expression analysis (e.g., emoji, image, logo) is used along with image detection, dimension reduction, and normalization. Feature extraction from a facial image and highlighting by classification (e.g., emotions, etc.) can also be implemented in the system.
Advantageously, illustrative embodiments provide domain-driven feedback data analysis with machine learning-based models and algorithms configured to learn the domain context in order to build a domain expertise corpus. More particularly, the analysis identifies specific domains related to feedback using artificial neural networks (ANNs) to learn from situations. ANNs are used to recognize and classify. Domain diagnosis analysis then detects the root cause using historical specific domain data.
Furthermore, illustrative embodiments provide a feedback analysis approach which trains the analyzer individually to support a specific domain. This is achieved by implementing functionality to build a knowledge repository as an expert system by learning over time in a domain context. This repository recommends the decisions regarding feedback and actions in that specific domain.
Still further, illustrative embodiments provide an inter-connected domain computing environment. One domain is interconnected with other dependent domains for data. This interconnectedness brings insights into other domains' decisions and actions to improve their findings and root causes. This ensemble approach can dynamically switch the context to all connected domains on an as-needed basis.
Further details of the various functionalities described above will be explained herein below in the context of
As mentioned, feedback domain analysis module 120 of domain-driven feedback data analysis system 100 is configured to analyze the context of feedback data and utilize a domain corpus to classify the feedback data into one or more domains. Feedback domain analysis module 120 uses NLP and neural networks to analyze the feedback data set and determine the intent.
Assuming that the feedback data set is similar to a time series model where the words come one after another in time/space, illustrative embodiments leverage a recurrent neural network (RNN). To better understand the context and analyze the feedback data set most efficiently, illustrative embodiments use a bi-directional RNN, which uses two separate processing sequences, one from left to right and another from right to left. As RNNs have the tendencies to explode or vanish with respect to gradient issues for longer and complex dialogs/messages, illustrative embodiments leverage a specific type of bi-directional RNN called a bi-directional RNN with LSTM (long short-term memory) for the NLP.
RNNs are a unique type of neural network where the previous step's output feeds into the current step's input. In a traditional neural network, which is also called a feed-forward network, input and output are independent. In language processing, it is important to remember the previous words before predicting the next term of a sentence. This is where an RNN makes a difference by having the hidden state recognize some phrases in the sentence. If the sentences are too long, some of that previous information may not be available in the limited hidden state, which requires the bi-directional processing of the sentence (from past and future in two sequences in a parallel manner) as provided by a bi-directional RNN. LSTM introduces advanced memory units and gates to an RNN to improve the model's accuracy and performance. A diagram of a bi-LSTM network with input, hidden, and output layers is shown as a machine learning model 300 in
The intent analysis starts with a set of corpus data (e.g., intent corpus 416) used to train the ML model 418. This corpus data contains the words and phrases and the intent associated with each sentence. First, the input feedback message 402 is pre-processed (e.g., text processing module 412) to clean any unwanted characters and stop words. This step may also include stemming and lemmatization, changing text to lowercase, removing punctuation, removing flawed characters, etc.
Once pre-processing and data cleanup is executed, the input list of words (in the sentences from the input feedback message 402) is tokenized. This is performed in feature engineering module 414. There are many approaches to tokenizing these words including, but not limited to, Keras library, NLTK library, etc. In one illustrative embodiment, a Keras tokenizer class is used to index the tokens. After tokenization is executed, they are padded to make them of equal length to be used in ML model 418.
For output encoding, the same is done (tokenization and padding) to the intent list of the corpus. At the end of this step, the intent list is indexed and ready to be fed into ML model 418. First, they need to be one-hot encoded before being provided to ML model 418.
As mentioned, the corpus containing intent data (416) is used to train ML model 418 before predicting the intent for incoming dialog. The bi-directional RNN model with LSTM (418) is created using the Keras library. Several parameters are passed during the creation of the model, including an Adam optimizer, an activation function such as Softmax, number of epochs, etc. These parameters, especially batch size and periods, are tuned to obtain the best performance and accuracy of the model. After the model is trained with the intent training data, it can predict the dialog message's intent (420). The accuracy of the model is calculated as well for hyperparameter tuning.
Sentiment analysis performed by sentiment classification engine 500 comprises understanding the feedback context and making appropriate decisions in a priority manner. The sentiment analysis follows the same pattern and algorithms as the intent analysis of intent classification engine 400, using a sentiment corpus (516) instead of an intent corpus. More particularly, the same pre-processing done in text processing module 412 is done in text processing module 512, and the same feature engineering done in feature engineering module 414 is done in feature engineering module 514, before training the same bi-LSTM network (ML model 418/518) for predicting the sentiment. As shown, there are two types of sentiment attributes identified, i.e., sentiment attributes 520 for how customers feel about a product, and sentiment attributes 522 for the product itself (e.g., for each item/part that made the feedback comments).
Recall that the feedback communication interface 110 of domain-driven feedback data analysis system 100 is the primary interface layer for communication with external system/network 102. Any inbound or outbound communication involving all types of messages by email, voice, and chat passes through feedback communication interface 110 before/after being processed.
In one illustrative embodiment, feedback communication interface 110 utilizes speech to text conversation. Messages received as audio/voice signals are converted to text before being analyzed for intent, sentiment, and contexts to make the root cause decisions and thus determine the most appropriate actions. This requires a speech to text model to convert a voice message to a text format. Illustrative embodiments assume that voice messages are received in digital signal format. If they are in analog format, sampling the signal includes converting it into a digital format. The sampling rate or sampling frequency is defined as the number of samples selected per second. The first step in speech recognition is to extract features from the voice signal, which will be input into the model later. There are multiple approaches to feature extraction including, but not limited to, time-domain analysis and frequency domain analysis. In the time-domain analysis, the voice signal is represented by the amplitude as a function of time, so essentially a plot between amplitude and time. In frequency domain analysis, it is a plot between frequency and amplitude. To represent features between frequency and time, a spectrograph can be plotted.
In some illustrative embodiments, TensorFlow is used which provides a speech command dataset that includes thousands of people's one-second long utterances of many words. This data set is used as training data for the created speech to text model. LibROSA and SciPy libraries can be used for audio processing in Python. Illustrative embodiments use neural networks for feature learning and prediction of audio to text. The speech-to-text model can use a one-dimension convolutional neural network called Conv1d, available in TensorFlow and Keras. In this neural network model, illustrated as a machine learning model 600 in
A voice (audio) signal 702 is received and pre-processed by audio processing module 712 by sampling (if needed) and removing any speech less than a given duration (e.g., one second). The sampled voice data is applied to the ML model 716, which is trained as follows. In a multi-classification use case, target value labels (which are categorical data) are encoded by a label encoder (e.g., classes available in the ScikitLearn library). Audio sample data from speech corpus 714 is broken into training (about 80%) and testing (about 20%) sets. ML model 716 is trained with the training data set. When ML model 716 is the machine learning model 600 shown in
Turning now to
More particularly, feedback data analysis decision engine 800 is responsible for deciding domain-related action from feedback. Decision engine 810 leverages an ensemble, decision tree-based bagging technique referred to as a random forest for multinomial classification of actions. The random forest model 812 uses historical training data containing multi-dimension data points for training. Once the random forest model 812 is fully trained, the conversation's (feedback data set's) state (i.e., intent, sentiment, and context) is used to predict an optimal or near optimal decision. Random forest model 812 uses a large group of complex decision trees and can provide classification predictions with a high degree of accuracy on any size of data. Decision engine 810 can predict the recommended virtual assistant with the accuracy or likelihood percentage. The accuracy of the random forest model 812 can be improved by hyperparameter tuning.
Illustrative embodiments are described herein with reference to exemplary information processing systems and associated computers, servers, storage devices and other processing devices. It is to be appreciated, however, that embodiments are not restricted to use with the particular illustrative system and device configurations shown. Accordingly, the term “information processing system” as used herein is intended to be broadly construed, so as to encompass, for example, processing systems comprising cloud computing and storage systems, as well as other types of processing systems comprising various combinations of physical and virtual processing resources. An information processing system may therefore comprise, for example, at least one data center or other type of cloud-based system that includes one or more clouds hosting tenants that access cloud resources. Cloud infrastructure can include private clouds, public clouds, and/or combinations of private/public clouds (hybrid clouds).
The processing platform 1000 in this embodiment comprises a plurality of processing devices, denoted 1002-1, 1002-2, 1002-3, . . . 1002-K, which communicate with one another over network(s) 1004. It is to be appreciated that the methodologies described herein may be executed in one such processing device 1002, or executed in a distributed manner across two or more such processing devices 1002. It is to be further appreciated that a server, a client device, a computing device or any other processing platform element may be viewed as an example of what is more generally referred to herein as a “processing device.” As illustrated in
The processing device 1002-1 in the processing platform 1000 comprises a processor 1010 coupled to a memory 1012. The processor 1010 may comprise a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other type of processing circuitry, as well as portions or combinations of such circuitry elements. Components of systems as disclosed herein can be implemented at least in part in the form of one or more software programs stored in memory and executed by a processor of a processing device such as processor 1010. Memory 1012 (or other storage device) having such program code embodied therein is an example of what is more generally referred to herein as a processor-readable storage medium. Articles of manufacture comprising such computer-readable or processor-readable storage media are considered embodiments of the invention. A given such article of manufacture may comprise, for example, a storage device such as a storage disk, a storage array or an integrated circuit containing memory. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals.
Furthermore, memory 1012 may comprise electronic memory such as random-access memory (RAM), read-only memory (ROM) or other types of memory, in any combination. The one or more software programs when executed by a processing device such as the processing device 1002-1 causes the device to perform functions associated with one or more of the components/steps of system/methodologies in
Processing device 1002-1 also includes network interface circuitry 1014, which is used to interface the device with the networks 1004 and other system components. Such circuitry may comprise conventional transceivers of a type well known in the art.
The other processing devices 1002 (1002-2, 1002-3, . . . 1002-K) of the processing platform 1000 are assumed to be configured in a manner similar to that shown for computing device 1002-1 in the figure.
The processing platform 1000 shown in
Also, numerous other arrangements of servers, clients, computers, storage devices or other components are possible in processing platform 1000. Such components can communicate with other elements of the processing platform 1000 over any type of network, such as a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or cable network, or various portions or combinations of these and other types of networks.
Furthermore, it is to be appreciated that the processing platform 1000 of
As is known, virtual machines are logical processing elements that may be instantiated on one or more physical processing elements (e.g., servers, computers, processing devices). That is, a “virtual machine” generally refers to a software implementation of a machine (i.e., a computer) that executes programs like a physical machine. Thus, different virtual machines can run different operating systems and multiple applications on the same physical computer. Virtualization is implemented by the hypervisor which is directly inserted on top of the computer hardware in order to allocate hardware resources of the physical computer dynamically and transparently. The hypervisor affords the ability for multiple operating systems to run concurrently on a single physical computer and share hardware resources with each other.
It was noted above that portions of the computing environment may be implemented using one or more processing platforms. A given such processing platform comprises at least one processing device comprising a processor coupled to a memory, and the processing device may be implemented at least in part utilizing one or more virtual machines, containers or other virtualization infrastructure. By way of example, such containers may be Docker containers or other types of containers.
The particular processing operations and other system functionality described in conjunction with
It should again be emphasized that the above-described embodiments of the invention are presented for purposes of illustration only. Many variations may be made in the particular arrangements shown. For example, although described in the context of particular system and device configurations, the techniques are applicable to a wide variety of other types of data processing systems, processing devices and distributed virtual infrastructure arrangements. In addition, any simplifying assumptions made above in the course of describing the illustrative embodiments should also be viewed as exemplary rather than as requirements or limitations of the invention.
