Patent Application
20220245212
Aspects of the disclosure generally relate to a semi-crowdsourced expert-in-the-loop method for extraction and compilation of diagnosis and repair knowledge.
Consumers often go to technicians with problems that are beyond the technicians' experience. When this occurs, the technicians consult a diagnosis-and-repair knowledge base.
In one or more illustrative examples, a method for semi-crowdsourced expert-in-the-loop information capture is provided. Solution search criteria for a repair problem are obtained from one or more of an expert user, machine algorithm, or crowd worker. A search is conducted according to the solution search criteria to identify search results. The search results are filtered according to relevance and likelihood of containing confirmed solutions using a confirmed-solution classifier. The search results are provided to crowd workers for analysis to find and extract confirmed solutions. If the solutions do not already exist in a knowledge base, the extracted confirmed solutions are provided for expert review and adding the extracted confirmed solutions, as reviewed, to the knowledge base. Otherwise if the solutions already exist in the knowledge base, the extracted confirmed solutions are used to retrain the confirmed-solution classifier.
In one or more illustrative examples, a system for semi-crowdsourced expert-in-the-loop information capture is provided. The system includes a computing platform including a hardware processor. The processor is programmed to obtain solution search criteria for a repair problem from one or more of an expert user, machine algorithm, or crowd worker; conduct a search according to the solution search criteria to identify search results; filter the search results according to relevance and likelihood of containing confirmed solutions using a confirmed-solution classifier; and provide the search results to crowd workers for analysis to find and extract confirmed solutions. The processor is programmed to, if the solutions do not exist in a knowledge base, provide the extracted confirmed solutions for expert review and adding the extracted confirmed solutions, as reviewed, to the knowledge base; and otherwise if the solutions already exist in the knowledge base, use the extracted confirmed solutions to retrain the confirmed-solution classifier.
In one or more illustrative examples, a non-transitory computer readable medium including instructions for semi-crowdsourced expert-in-the-loop information capture is provided. When the instructions are executed by a processor of a computing device, the computing device is programmed to obtain solution search criteria for a repair problem from one or more of an expert user, machine algorithm, or crowd worker; conduct a search according to the solution search criteria to identify search results; filter the search results according to relevance and likelihood of containing confirmed solutions using a confirmed-solution classifier; provide the search results to crowd workers for analysis to find and extract confirmed solutions; if the solutions do not exist in a knowledge base, provide the extracted confirmed solutions for expert review and adding the extracted confirmed solutions, as reviewed, to the knowledge base; and otherwise if the solutions already exist in the knowledge base, use the extracted confirmed solutions to retrain the confirmed-solution classifier.
As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.
When domain experts compile a problem-solution knowledge base for diagnosis and repair, they are usually given a list of problems for which they conduct extensive online search to find valid solutions. A solution is deemed valid if someone confirms that it solved the problem. For example, when given a problem such as “Honda Accord shakes when going above 50 MPH”, the domain expert may search online web forums for discussions regarding this problem, read the discussion posts, and extract posts which contain solutions that are confirmed by at least one poster to have worked. Such a post would be considered a “Confirmed Solution”, and an example may be: “I had this same problem, and I replaced my carburetor and that fixed it.” The domain expert then includes this solution as an official, confirmed solution to the problem in the repair knowledge base.
Aspects of the disclosure generally relate to the semi-crowdsourced expert-in-the-loop extraction and compilation of diagnosis and repair knowledge. Given a repair problem, solution search criteria are obtained from one or more of an expert user, machine algorithm, or crowd worker. A search is then conducted, where the results are filters based on relevance and likelihood of containing confirmed solutions (e.g., using a confirmed-solution classifier). Crowd workers analyze the search results to find and extract confirmed solutions. The extracted solutions are checked to see if they already exist in a knowledge base. If the solutions don't exist, experts review the new solutions and add them to the knowledge base accordingly. Otherwise, the solutions are simply used to retrain the confirmed-solution classifier.
This method reduces cost by saving domain experts time in the search for solutions for inclusion in the knowledge base. The method achieves time saving by leveraging machine and low-cost crowd workers in the search effort; the domain experts need only to provide minimum initial search guidance and the final review and approval of extracted solutions. The bulk of the search and solution-extraction work is done by machine and crowd workers.
When domain experts read web forum posts to find solutions, they rely on sentence sentiment (e.g., phrasing) to identify confirmed solutions. The domain experts may identify key phrases such as “I had this same problem, and I did this to fix it” or “In my experience, these steps solved the problem” and so forth. This particular sentiment analysis can be performed by non-domain experts such as crowd workers, who can perform this task at lower cost than domain experts. However, while the crowd workers can understand sentence sentiment, the crowd workers may lack domain expertise and thus may be unable to correctly perform the initial search for sources related to the repair problem.
Thus, the disclosed approach combines the strengths of both the domain experts and crowd workers to extract solutions and compile knowledge bases at lower cost and at scale. The domain expert provides guidance for initial search, the machine performs the search and relays the results to the crowd workers for sentiment analysis to find confirmed solutions, and the domain expert then quickly reviews the extracted solutions and incorporates them into the knowledge base.
The knowledge that is learned from crowd and the expert is used to improve machine learning models. Small datasets initially created via crowdsourcing and/or domain expert get increased with various methods and new knowledge generated by the system. The model performance is improved with new datasets that are injected to the model and the system that creates crowdsourcing tasks and expert review tasks act like an active learning.
At operation 204, domain experts provide initial guidance to machine for conducting an online solution search. This guidance may include, as some examples: product of interest details (e.g., car make, model, year, engine type), search terms (inclusion criteria) for the repair problems (e.g., “shaking above 35 mph”), and/or exclusion criteria aspects which are commonly found through the inclusion criteria but that the experts know are not relevant (e.g., “intake valve”).
In the alternative, at operation 206 machine strategy is utilized through leveraging domain vocabularies and/or domain models (e.g., car component dictionaries, action verb, etc.). As yet a further alternative, at operation 208, crowd workers may read the repair problem and form a search strategy (e.g., create a set of appropriate Google search terms). It should be noted that a combination of operations 204, 206, and 208 may be performed. As one combined possibility, the machine-generated search terms can also be optionally reviewed by experts. As another combined possibility, crowd-sourced search terms may be evaluated by machine in order to discard terms with low quality or relevance.
Regardless, from operations 204, 206 and 208, control proceeds to operation 210 at which a system 100 conducts searches based on product (e.g., vehicle) information and search terms provided by a domain expert along with machine support as described in operations 204, 206, and 208. Further the system 100 may filter the results for ones with high relevance to the original repair problem and from forums.
These searches may be performed by the parser 108, using various search APIs (e.g., Google(™) Search API, Bing(™) Search API, etc.). For instance, the searches may be used to retrieve a set of top K URLs (e.g., K=30 URLs in one example). Web page types of interest may then be selected from the retrieved URLs. In an example, the URLs may be selected as being web discussion forums, but other examples are possible. The experience-based repair knowledge may require a confirmation of a fix for a proposed repair solution. Therefore, the original source should have an indication on “confirmed-fix” in comments in a web source provided by users or forum posting.
The system 100 may utilize the forum classifier 110 to perform forum classification based on rules and/or a machine learning model trained with positive and negative forum sources. Examples of heuristic rules include forum related keywords in URL and content, frequency of date and time format in the content, pagination in the content, user information in the content, and/or user likes in the content. These heuristics can be updated over time. Some examples of the heuristic rules are illustrated in
The forum classifier 110 may also rank sources with high relevance. This ranking may include, for example, matching the search criteria (e.g., product information, problem, product components in problem (e.g., my engine is loud if a car problem) etc.) This ranking may also include using domain knowledge about products to do smarter searching/filter (e.g., identifying that since two vehicle models have the same engine, the system 100 may search for both when given one or the other). As another possibility, the ranking may include using a component dictionary to conduct searches with synonyms, including informal terms for product parts. The system 100 may also use a repair action vocabulary and/or dictionary that contains the action/verb/adjective synonyms to expand the search, e.g., “fix/repair”, “replace/change”, “loud/noisy”, etc. Different weights are applicable to the presence/location of relevant words in the HTML content. Based on the scores from product relevance and search term relevance, the sources may be sorted.
Presence of problem post, solution post and confirmed-fix solutions as detected by machine learning, using the confirmed-fix classifier 112. If the confirmed-fix probability score of a given web resource is above a threshold (e.g., 90%) and/or below a lower bound (e.g., 10%), then the operation 212 with crowd workers can be skipped. The source with a positive score is presented to the expert and the source with a negative score is discarded.
Construction of grounds truth for confirmed fix classifiers may be provided as follows, using the confirmed fix human task processor 114. Each post extracted from a web resource may be used to classify problem post, solution post and confirmed-fix post. An initial dataset may be constructed with crowdsourcing or domain expert's annotation. As for the problem post, the first post of each forum thread may be used. The initial dataset may contain small positive confirmed-fix posts where each post indicates positive sentiment about fixing a problem with a proposed solution. Note that majority of posts in the web sources often contain non-positive posts. The solution posts may be first identified by searching posts that contain/matches with terminologies of product parts in the domain component dictionary. The matching posts may be used for crowdsourcing or domain expert's annotation. Based on this initial dataset, additional keywords for positive confirmed fix post may be constructed by leveraging synonyms of keywords found in the positive post. For example, if a post includes language such as “it solved my problem,” “work out,” and “resolve” are added as synonym keywords for a keyword of “solve”. Also, search/crawl/parse web sources may be used to find posts that contain keywords and terminologies in components dictionary and create crowdsourcing tasks to increase confirmed fix and solution positive posts in the dataset.
Training of the confirmed fix classifier 112 model may be performed as follows. The models may be trained with both positive and negative posts. Various machine learning algorithms such as multi-layer perception (MLP), random forest, logistic regression and deep learning classification models may be used for training models. Features used in the machine learning models may be vectorized with term frequency-inverse document frequency (TFIDF), a dependency parser and/or use of N-gram parsing, but these are non-limiting examples and the possibilities are not limited to those approaches. For sources with high confirmed-fix probability, the system 100 may use adjacent posts (to the confirmed-fix post) to classify solution post or not. Based on the score of product problem post prediction, confirmed fix post prediction and solution post prediction, the total prediction of a given resource may be estimated. Based on the prediction for a given resource, the system 100 determines if a resource is need for a crowdsourcing task or not.
At operation 212, crowd workers analyze the search results filter by operation 210 to find and extract solutions. This may be accomplished, for example, via the confirmed fix human task processor 114 interacting with the user interfaces 102. For example, the search results may be a series of web forums discussing repair problems similar to the given problem, and crowd workers read the posts and find confirmed solutions based on the sentiment of the text (e.g., “I had then same problem, and this is how I fixed it . . . ”). An example template user interface for crowd workers to find and extract confirmed solutions from a machine-located forum thread is discussed below with respect to
The crowd workers may verify that the search result web page is a source of repair information, e.g., a web forum thread discussing a particular car repair problem (examples of which are shown in
At operation 214, the system 100 uses a similarity computation to check whether the problem and solution combination already exists in the knowledge base. If not, then an expert may confirm and add it to the knowledge base 122 at operation 216. Otherwise, the results may be used only for improving the confirmed-fix classifier 112 at operation 218. Thus, at operation 218, the model performance is improved with new datasets that are injected into the confirmed-fix classifier 112 model such that the system 100 creates crowdsourcing tasks and expert review tasks to perform active learning.
At operation 308, the forum classifier 110 utilizes rule-based forum scoring to score the top URLs. These rules 310 may include for example, applying scoring based on various aspects of the content of the webpage of the URL. As some examples, if a forum-related keyword is located in the URL itself a first value may be applied to the URL (e.g., 0.5), if a forum-related keyword is located in the content of the webpage a second value may be applied to the URL (e.g., 0.2), if at least a frequency of data and time formats are found in the content a third value may be applied to the URL (e.g., 0.1), if pagination is found in the content a fourth value may be applied to the URL (e.g., 0.05), if user information is found in the content a fifth value may be applied to the URL (e.g., 0.1), and/or if user likes are found in the content a sixth value may be applied to the URL (e.g., 0.05).
At operation 312, using the scoring rules, a list of the URLs is compiled. For instance, this list may include the URLs having the highest scores as determined at operation 308 using the rules 310. Further aspects of the URL scoring are described in
At operation 314, the forum classifier 110 crawls the URLs of the list to retrieve HTML content 316 from the URLs. The body content is extracted at operation 318 (e.g., from the HTML markup into plain text without markup).
At operation 320, each post within the extracted body content is extracted from the web page. For instance, a database of stop words 322 may be used to determine post boundaries in situations where at least a predefined number of the stop words (e.g., 3) are found in the post at operation 324. Based on each divided out post, at operation 326 each post is queried back into the HTML content to extract the post accurately. For example, this may involve retrieving the entire post as a separate web result.
At operation 328 the posts from the URLs are listed as output. Also, at operation 330, the URL content of the posts is also crawled for additional links that may be used to further inform the find operation.
With respect to the brand or make 402, if at operation 414 any of these portions of the crawl content 412 include the brand or make 402 then the score for that URL may be increased at operation 416 (e.g., by ⅓ on a scale of 0 to 1). With respect to the model 404, if at operation 418 any of these portions of the crawl content 412 include the model 404 then the score for that URL may be increased at operation 420 (e.g., by ⅓ on a scale of 0 to 1). With respect to the search terms 406, if at operation 422 any of these portions of the crawl content 412 include at least partial (e.g., >=0.5) search terms 406 then the score for that URL may be increased at operation 424 (e.g., by ⅓ on a scale of 0 to 1). Or, if at operation 426 the body content includes at least partial (e.g., >=0.6) search terms 406 and other sections have a merged total score greater than zero, the score for that URL may also be increased at operation 424. At operation 426, the total score from each of the operations 416, 420, and 424 is totaled to determine the total score (e.g., on a scale of 0 to 1).
Thus, by using the describes systems and method, semi-crowdsourced expert-in-the-loop information capture is provided. Solution search criteria for a repair problem are obtained from one or more of an expert user or a machine algorithm. A search is conducted according to the solution search criteria to identify search results. The search results are filtered according to relevance and likelihood of containing confirmed solutions using a confirmed-solution classifier. The search results are provided to crowd workers for analysis to find and extract confirmed solutions. If the solutions do not exist in a knowledge base, the extracted confirmed solutions are provided for expert review and adding the extracted confirmed solutions, as reviewed, to the knowledge base. Otherwise if the solutions already exist in the knowledge base, the extracted confirmed solutions are used to retrain the confirmed-solution classifier.
The algorithms and/or methodologies of one or more embodiments discussed herein are implemented using a computing device, such as the computing platform 700 illustrated in
The memory 702 may include a single memory device or a number of memory devices including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. The non-volatile storage 704 may include one or more persistent data storage devices such as a hard drive, optical drive, tape drive, non-volatile solid-state device, cloud storage or any other device capable of persistently storing information.
The processor 706 may include one or more devices selected from high-performance computing (HPC) systems including high-performance cores, microprocessors, micro-controllers, digital signal processors, microcomputers, central processing units (CPU), graphical processing units (GPU), tensor processing units (TPU), field programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that manipulate signals (analog or digital) based on computer-executable instructions residing in memory 702.
The processor 706 may be configured to read into memory 702 and execute computer-executable instructions residing in the non-volatile storage 704. Upon execution by the processor 706, the computer-executable instructions may cause the computing platform 700 to implement one or more of the algorithms and/or methodologies disclosed herein.
The computing platform 700 may further include one or more input devices 708, such as buttons and/or touch-sensitive displays screens, and output devices 710 such as lights, speakers, and/or display screens. The computing platform 700 may also include one or more network devices 712, such as modems or other wired or wireless transceivers that may be used to allow the computing platform 700 to communicate with other computing platforms 700 over a communications network.
Computer-readable program instructions stored in a computer readable medium may be used to direct a computer, other types of programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions that implement the functions, acts, and/or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, acts, and/or operations specified in the flowcharts and diagrams may be re-ordered, processed serially, and/or processed concurrently consistent with one or more embodiments. Moreover, any of the flowcharts and/or diagrams may include more or fewer nodes or blocks than those illustrated consistent with one or more embodiments.
While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms of the invention. Rather, the words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the invention. Additionally, the features of various implementing embodiments may be combined to form further embodiments of the invention.
