Patent Grant
7653611
The subject invention relates generally to data mining, and more particularly to systems and methods for utilizing unobtrusive logging of an entity's response to diagnostic reports to facilitate in improving diagnostic reports.
Computers have become an integral part of society. Every day people become more dependent on this technology to facilitate both work and also leisure activities. They typically consist of “hardware” and “software.” The hardware is the physical components of the computer while the software constitutes “programs” that are run on the computer. Examples of some common programs include word processors, spreadsheets, and email programs. In general, a user can typically use these programs to some extent. Basic features might be intuitive and require little training to use. However, at some point, the user of the program will run into a task that they are unable to complete without help. Rather than provide a “live” operator to answer questions, most software programs include an “assistant” that helps the user when they become stuck on a problem. General forms of an assistant include help indexes with search support, common problems encountered, and/or user specific solutions and the like.
An effective way of building the assisting program is using a platform commonly known as an “assistant platform.” The assistant platform or “AP” typically provides a user interface that allows the user to look up answers and/or actually pose questions so that the AP can provide an answer. During these user queries, data is collected by the AP as to the adequacy and/or relevancy and the like of the AP response to a user's query. This information is then analyzed, compiled, and generated into diagnostic reports that contain one or more suggestions for improving the AP. The AP typically has a search authoring tool user interface that allows an author of the software program to retrieve the diagnostic reports. The author desires to improve the AP so that a user will become more satisfied with it and not become frustrated by unhelpful suggestions to a problem. Thus, the author reviews the diagnostic reports and their suggestions. It is possible that the author may encounter several hundred or even thousand diagnostic reports for a popular program utilized by many users. The author typically must sort through the list of reports and determine which reports are worth reviewing and taking actions on. This is a very time consuming task and is often accomplished using a computing device so that the author can remark each suggestion.
It is easily imaginable that if the diagnostic list is extremely long, that some reports will not be reviewed and that insignificant suggestions or incorrect suggestions might be reviewed in detail before they are properly assessed, substantially reducing the author's productivity and performance. The difficulty of the reviewing process is further compounded by the fact that in some situations the proper content existed in the AP, but the user's intent was misinterpreted, while in other situations, the user's intent was properly assessed but the correct information was not available to the AP. Thus, different situations can require different solutions to resolve. For these reasons, difficult and time consuming situations may be put aside until the easier suggestions are implemented first. Therefore, the author's workload is dictating priority rather than a true priority based on the significance of the suggestions provided by the AP.
The result of large quantities of suggestions and complex user intent problems is a poorly improved AP. This causes an increase in user dissatisfaction with the software product, possibly generating even more diagnostic reports and/or decreasing sales of the software product itself. Unfortunately, this is a typical scenario for many APs and their authors. Similar problems can also occur with non-typical software programs as well. For example, it is common today to find computers and software in automobiles as well as other machinery. These types of software can also generate diagnostic reports that need to be reviewed by an author as well. Thus, as programs become even more sophisticated, the need for assistance to the user and/or even to other interfacing devices (e.g., an artificial intelligence device and the like) will only increase as well, burdening reviewing entities even more and greatly decreasing user satisfaction. If a business is dependent on the proper operation of a program, the user's inability to make the program perform effectively could also severely impact business' profits.
The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key/critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.
The subject invention relates generally to data mining, and more particularly to systems and methods for utilizing an entity's response to diagnostic reports in order to facilitate in improving diagnostic reports. Data logging of responses to diagnostic reports is leveraged to provide data that can be mined for diagnostic report quality information. Instances of the subject invention provide an initial diagnostic report assessment means to facilitate review by an entity. The entity's responses to the sorted diagnostic reports are logged unobtrusively to create diagnostic report quality data. This data is then analyzed by an analysis means that can then adjust the assessment means to improve its performance. Other instances of the subject invention facilitate to increase the performance of a diagnostic report generating means as well. Instances of the subject invention can also employ machine learning techniques to facilitate in analyzing the quality data and/or in assessing the diagnostic reports.
Consequently, instances of the subject invention provide a means to increase performance of diagnostic report assessment models and/or diagnostic report generating systems via analysis of the mined quality information. This allows the workload of an entity charged with reviewing the diagnostic reports to be substantially reduced while performance is increased. Moreover, a user interfacing with an assisting mechanism such as, for example, an assistant platform, can have increased user satisfaction because the entity, such as, for example, an author of a software product, is able to provide a higher performance assisting mechanism with less effort. Thus, instances of the subject invention significantly increase productivity of the entity while simultaneously increasing performance of the assisting mechanism.
To the accomplishment of the foregoing and related ends, certain illustrative aspects of the invention are described herein in connection with the following description and the annexed drawings. These aspects are indicative, however, of but a few of the various ways in which the principles of the invention may be employed and the subject invention is intended to include all such aspects and their equivalents. Other advantages and novel features of the invention may become apparent from the following detailed description of the invention when considered in conjunction with the drawings.
The subject invention is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject invention. It may be evident, however, that the subject invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the subject invention.
As used in this application, the term “component” is intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a computer component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. A “thread” is the entity within a process that the operating system kernel schedules for execution. As is well known in the art, each thread has an associated “context” which is the volatile data associated with the execution of the thread. A thread's context includes the contents of system registers and the virtual address belonging to the thread's process. Thus, the actual data comprising a thread's context varies as it executes.
Instances of the subject invention provide systems and methods that improve diagnostic techniques. These include, but are not limited to, content and/or search engine diagnostic techniques that provide self improvement. This can be accomplished, for example, by logging authors' usage data with regard to diagnostic reports which facilitate in deriving the authors' feedback on the quality of the reports, and the quality of the internal measurements of a user search such as, for example, user search bundles and intent bundles. Learning models, for example, can then be constructed to predict in what situations the measurements work well and in what situations the measurements do not work well. These models can then be applied to improve the authors' productivity through filtering out low quality reports and/or components. These models can also be utilized to improve the diagnostic technology and systems themselves.
In general, diagnostic reports are generated by an assisting mechanism that facilitates users who seek additional information about an associated product. In one instance of the subject invention, an assistant platform (AP) associated with a software product (e.g., a word processor, accounting software, spreadsheet, etc.) generates diagnostic reports from data obtained from user queries regarding the software product. The diagnostic reports are assessed via criteria such as, for example, highest-to-lowest confidence techniques and the like. The assessment allows an author associated with the software product to address higher priority suggestions from the diagnostic reports first, increasing the author's productivity. As the author interacts with the suggestions, their responses are unobtrusively logged and analyzed to provide improvements in assessing the diagnostic reports. In other instances of the subject invention, improvements can also be made to the assistant platform itself to facilitate in obtaining improved suggestions in the diagnostic reports.
Although instances of the subject invention can be utilized with typical software products, they can also be utilized with specialized type software products in devices other than a common computer. These include, but are not limited to, vehicle diagnostic systems; industrial machinery diagnostic systems; heating, ventilation, and air conditioning (HVAC) diagnostic systems, heavy equipment diagnostic systems, and electronic device diagnostic systems and the like. Thus, for example, instances of the subject invention can be utilized with automobile on-board diagnostic systems to facilitate in improving the diagnostic software that aids a mechanic (i.e., user) in diagnosing automobile problems and the like.
In
The diagnostic report improvement component 102 assesses the diagnostic reports 106 and interacts with the entity 104 to enhance the entity's work performance and to obtain work product logs associated with suggestions from the diagnostic reports 106. The work product logs are obtained unobtrusively from the interaction between the entity 104 and the diagnostic report improvement component 102. The logs are then analyzed to facilitate in improving the assessment technique utilized when the diagnostic reports 106 are received. In this manner, the productivity of the entity 104 is increased due, in part, to the appropriate priority lists being generated through the assessment process. This allows the most productive diagnostic reports to be reviewed by the entity 104 first, increasing the effectiveness of the entity's diagnostic report reviewing efforts. In other instances of the subject invention, the diagnostic report improvement component 102 can also provide an optional improved quality model 108 that can be utilized to improve diagnostic report generation and the like. The improved quality model 108 can also include a quality report that details the analysis of the logged entity responses in relation to the effectiveness of the diagnostic reports. One skilled in the art can appreciate that in some instances of the subject invention, the assessment process can be initially equivalent to passing the diagnostic reports to the author unchanged to facilitate in providing initial data to improve the diagnostic reports via the diagnostic report improvement component 102. Likewise, the initial diagnostic reports can be directly accessed by the entity 104, bypassing the diagnostic report improvement component 102 altogether.
Turning to
In other instances of the subject invention, the analysis component 214 facilitates the diagnostic report generating system 206 in improving the diagnostic reports 204 themselves. Thus, the analysis component 214 can interact with the diagnostic report generating system 206 directly and/or via human interaction and the like (e.g., artificial intelligence, etc.). In still other instances of the subject invention, the diagnostic report improvement component 202 can facilitate in improving both the diagnostic reports 204 and also their assessment by the assessment model component 210. The analysis component 214 can also provide an improved quality model 216 that can include a quality report with regard to the helpfulness of the diagnostic report suggestions as determined by the entity 208. The improvements in reporting and/or assessing allow the subject invention to substantially increase the efficiency and productivity of the entity 208. This provides a better assisting mechanism for a device and/or product and typically increases user satisfaction of that assisting mechanism. Users tend to associate all aspects of a device and/or product as a single item so that by increasing the helpfulness of the assisting mechanism, the users have increased satisfaction and productivity with regard to the overall product and/or device as well. This is typically the result of the fact that one purpose of the assisting mechanism is to help a user better utilize the device and/or product.
Looking at
The AP 316 is typically found with software products to assist a software product user when the user needs additional information to facilitate use of the software product. Thus, a user interacts with the AP 316 and provides query user data 318 that is received by the AP's client 320 which is then relayed to the AP server 322. The AP analysis mechanism 324 then retrieves the data from the AP server 322 and analyzes the query user data 318 to facilitate in providing diagnostic reports. The author 306 can then interact with the search authoring tool UI 326 to request the diagnostic reports 304 based on the query user data 318. The assessment model component 308 then receives the diagnostic reports 304 and assesses an aspect of the diagnostic reports 304 to facilitate the author 306 in reviewing them 304. The aspect can include, but is not limited to, confidences associated with each of the diagnostic reports 304 and the like. The author 306 can then review the assessed diagnostic report list and respond to the diagnostic report suggestions. The workflow logging component 310 logs the author's responses in an unobtrusive manner. The logged data is then analyzed by the analysis component 312 to facilitate in determining the quality of the diagnostic reports 304 and to facilitate in improving the assessment model component 308.
In other instances of the subject invention, the analysis component can also provide a quality model that can, for example, include a quality report with regard to the diagnostic reports 304. In still other instances of the subject invention, the analysis component 312 can also facilitate in improving the AP analysis mechanism 324 of the AP 316. The analysis component 312 can interact with the AP analysis mechanism 324 directly and/or via human interaction and the like (e.g., artificial intelligence, etc.). This allows the AP analysis mechanism 324 to produce a higher quality diagnostic report. Thus, the combination of improving both the generated diagnostic reports and also the assessment of those improved reports provides a substantial improvement in diagnostic report reviewing by the author 306. This greatly increases the efficiency and productivity of the author 306 without interference. Hence, instances of the subject invention can be implemented with existing workflow and AP assisted products without requiring additional training and/or causing workflow impairment.
Consequently, instances of the subject invention can provide improved systems and methods of generating and/or assessing diagnostic reports to improve help systems via data mining techniques that facilitate improvements in, for example, diagnostic report generation for assistant systems. For example, instances of the subject invention can utilize processes to provide improvements for an assistant system of a software product by generating diagnostic reports through analyzing end user usage and/or satisfaction data logs from the assistant system. These reports can present to the authors different types of problems occurring in the assistant system, and suggest corresponding actions to correct the problems. The types of problems can include scenarios such as: a) end users issued a set of similar queries but did not get any satisfied results back from the system, suggesting authors to write new assets related to these queries and b) end users issued a set of similar queries but did not get any satisfied results back directly, however through sequential analysis, the reports suggests relevance assets for authors to map such that after taking these new mappings, the search engine can give relevant results when these queries are issued again later.
Unfortunately, these reports are not always accurate. Thus, the data mining techniques provided by instances of the subject invention can improve the diagnostic report generation. For example, one technique utilizes several processes including data logging. The data logging allows feedback to be gathered from authors on the quality of each diagnostic report. This can be accomplished by building a data log into the authors' work flow. When an author takes the suggested action of a report, the suggestion can be logged, for example, as a good suggestion. Otherwise, when an author discards the suggested action of a report, the suggestion can be logged, for example, as a bad suggestion. However, if the authors take no action, it is typically indeterminate because the authors could have taken no action due to a bad suggestion or taken no action because they desire to review it at a later time. Likewise, a suggestion might not be totally good or totally bad. The author can dislike the suggestion but agree that a problem exists and needs to be fixed in a different manner. The author can also disagree with a problem statement but agree that a fix is needed. In other instances of the subject invention, even a discarded suggestion can be deemed indeterminate. One skilled in the art can appreciate that other variations exist and are within the scope of the subject invention.
The good and bad suggestions mined from the data logs can then be utilized to construct a model to facilitate in predicting what kind of suggestions in diagnostic reports are good and what kinds of suggestions are bad. This model can then be utilized to sort diagnostic reports and suggestions in the reports so that the authors can work on high quality suggestions first. This can significantly increase the authors' productivity. Additionally, the model can be analyzed, for example, to identify in what situations the suggestions are bad such that ways can be found to improve the diagnostic report generation.
In view of the exemplary systems shown and described above, methodologies that may be implemented in accordance with the subject invention will be better appreciated with reference to the flow charts of
The invention may be described in the general context of computer-executable instructions, such as program modules, executed by one or more components. Generally, program modules include routines, programs, objects, data structures, etc., that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various instances of the subject invention.
In
Turning to
In order to provide additional context for implementing various aspects of the subject invention,
As used in this application, the term “component” is intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and a computer. By way of illustration, an application running on a server and/or the server can be a component. In addition, a component may include one or more subcomponents.
With reference to
The system bus 608 may be any of several types of bus structure including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of conventional bus architectures such as PCI, VESA, Microchannel, ISA, and EISA, to name a few. The system memory 606 includes read only memory (ROM) 610 and random access memory (RAM) 612. A basic input/output system (BIOS) 614, containing the basic routines that help to transfer information between elements within the computer 602, such as during start-up, is stored in ROM 610.
The computer 602 also may include, for example, a hard disk drive 616, a magnetic disk drive 618, e.g., to read from or write to a removable disk 620, and an optical disk drive 622, e.g., for reading from or writing to a CD-ROM disk 624 or other optical media. The hard disk drive 616, magnetic disk drive 618, and optical disk drive 622 are connected to the system bus 608 by a hard disk drive interface 626, a magnetic disk drive interface 628, and an optical drive interface 630, respectively. The drives 616-622 and their associated computer-readable media provide nonvolatile storage of data, data structures, computer-executable instructions, etc. for the computer 602. Although the description of computer-readable media above refers to a hard disk, a removable magnetic disk and a CD, it should be appreciated by those skilled in the art that other types of media which are readable by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like, can also be used in the exemplary operating environment 600, and further that any such media may contain computer-executable instructions for performing the methods of the subject invention.
A number of program modules may be stored in the drives 616-622 and RAM 612, including an operating system 632, one or more application programs 634, other program modules 636, and program data 638. The operating system 632 may be any suitable operating system or combination of operating systems. By way of example, the application programs 634 and program modules 636 can include a diagnostic report improvement scheme in accordance with an aspect of the subject invention.
A user can enter commands and information into the computer 602 through one or more user input devices, such as a keyboard 640 and a pointing device (e.g., a mouse 642). Other input devices (not shown) may include a microphone, a joystick, a game pad, a satellite dish, a wireless remote, a scanner, or the like. These and other input devices are often connected to the processing unit 604 through a serial port interface 644 that is coupled to the system bus 608, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). A monitor 646 or other type of display device is also connected to the system bus 608 via an interface, such as a video adapter 648. In addition to the monitor 646, the computer 602 may include other peripheral output devices (not shown), such as speakers, printers, etc.
It is to be appreciated that the computer 602 can operate in a networked environment using logical connections to one or more remote computers 660. The remote computer 660 may be a workstation, a server computer, a router, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 602, although for purposes of brevity, only a memory storage device 662 is illustrated in
When used in a LAN networking environment, for example, the computer 602 is connected to the local network 664 through a network interface or adapter 668. When used in a WAN networking environment, the computer 602 typically includes a modem (e.g., telephone, DSL, cable, etc.) 670, or is connected to a communications server on the LAN, or has other means for establishing communications over the WAN 666, such as the Internet. The modem 670, which can be internal or external relative to the computer 602, is connected to the system bus 608 via the serial port interface 644. In a networked environment, program modules (including application programs 634) and/or program data 638 can be stored in the remote memory storage device 662. It will be appreciated that the network connections shown are exemplary and other means (e.g., wired or wireless) of establishing a communications link between the computers 602 and 660 can be used when carrying out an aspect of the subject invention.
In accordance with the practices of persons skilled in the art of computer programming, the subject invention has been described with reference to acts and symbolic representations of operations that are performed by a computer, such as the computer 602 or remote computer 660, unless otherwise indicated. Such acts and operations are sometimes referred to as being computer-executed. It will be appreciated that the acts and symbolically represented operations include the manipulation by the processing unit 604 of electrical signals representing data bits which causes a resulting transformation or reduction of the electrical signal representation, and the maintenance of data bits at memory locations in the memory system (including the system memory 606, hard drive 616, floppy disks 620, CD-ROM 624, and remote memory 662) to thereby reconfigure or otherwise alter the computer system's operation, as well as other processing of signals. The memory locations where such data bits are maintained are physical locations that have particular electrical, magnetic, or optical properties corresponding to the data bits.
It is to be appreciated that the systems and/or methods of the subject invention can be utilized in diagnostic report improvement facilitating computer components and non-computer related components alike. Further, those skilled in the art will recognize that the systems and/or methods of the subject invention are employable in a vast array of electronic related technologies, including, but not limited to, computers, servers and/or handheld electronic devices, and the like.
What has been described above includes examples of the subject invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the subject invention, but one of ordinary skill in the art may recognize that many further combinations and permutations of the subject invention are possible. Accordingly, the subject invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5307262 | Ertel | Apr 1994 | A |
| 5537590 | Amado | Jul 1996 | A |
| 5701400 | Amado | Dec 1997 | A |
| 6434572 | Derzay et al. | Aug 2002 | B2 |
| 6509914 | Babula et al. | Jan 2003 | B1 |
| 6553372 | Brassell et al. | Apr 2003 | B1 |
| 6598011 | Howards Koritzinsky et al. | Jul 2003 | B1 |
| 6643639 | Biebesheimer et al. | Nov 2003 | B2 |
| 6832199 | Kucek et al. | Dec 2004 | B1 |
| 6936476 | Anderson et al. | Aug 2005 | B1 |
| 7007074 | Radwin | Feb 2006 | B2 |
| 7255406 | Huck et al. | Aug 2007 | B1 |
| 7263710 | Hummel et al. | Aug 2007 | B1 |
| 20030055321 | Watrous et al. | Mar 2003 | A1 |
| 20030061071 | Babula et al. | Mar 2003 | A1 |
| 20040199332 | Iliff | Oct 2004 | A1 |
| 20050086195 | Tan et al. | Apr 2005 | A1 |
| 20050125390 | Hurst-Hiller et al. | Jun 2005 | A1 |
| 20050222901 | Agarwal et al. | Oct 2005 | A1 |
| 20050222981 | Lawrence et al. | Oct 2005 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20060241908 A1 | Oct 2006 | US |
