Patent Application
20070300215
Preferred embodiments of the subject matter described herein will now be explained with reference to the accompanying drawings of which:
Service provider 104 may distribute software updates to software hosts and may assign a unique identifier to each software update. The identifier may be any suitable identifier that uniquely identifies a software update. Examples of update identifiers that may be used include an application name for new applications or an application name plus a version number for upgrades to existing applications. Software hosts may also be uniquely identified by hostname or IP address. The unique identifiers for software updates and software hosts may be recorded by update provider 108 and used in update event records, which will be described in more detail below.
In order to monitor the performance effects of software update 102 on software host 100, a vetting score client 110 may reside on software host 100. Vetting score client 110 may be downloaded along with software update 102. Alternatively, vetting score client 110 may be obtained from a storage medium associated with software update 102 or from a separate download or storage medium. In alternate implementations, vetting score client 110 may be installed on a centralized platform maintained by the software update provider or by a third party.
Vetting score client 110 may collect identification information regarding the software update and the software host. This identification information may include any of the software update or software host identifying information described above. In order to collect this information, vetting score client 110 may include a watchdog function that monitors system logs to determine whether software has been installed. Examples of system logs that may be monitored include the UNIX SYSLOG and the Windows Event View Application/System logs. An example of a watchdog application that may be used to monitor these logs is the SWATCH program available at http://swatch.sourceforge.net/.
Vetting score client 110 may monitor a plurality of different performance-related parameters for the software host and determine point values by comparing each monitored performance parameter to a corresponding scoring rule that converts each monitored performance parameter into a point value. The point values may be combined into a score indicative of an overall performance of the effect of software update 102 on software host 100. Vetting score client 110 may also create and store an update event record, which creates an association between the software update, the host, and the score.
In order to monitor performance parameters, a performance monitor 112 may obtain performance-related parameters 114 to be monitored. Exemplary performance-related parameters that may be monitored include CPU usage, communications availability, trouble ticket generation, and availability of other applications on software host 100. Additional examples of performance-related parameters will be described below. In one example, vetting score client 110 may include a user interface 115 for allowing the user to select or otherwise indicate performance-related parameters to be monitored. In an alternate example, performance monitor 112 may be adapted to download performance-related parameters to be monitored from service provider 104 or from a standards organization. In yet another alternate example, performance monitor 112 and/or software update 102 may be preconfigured with performance-related parameters to be monitored.
Performance monitor 112 may obtain values for the performance-related parameters from a performance data source 116. Performance data source 116 may include local and/or remote utilities for monitoring performance of software host 100. Examples of local performance data sources include command line utilities available to the operating system of software host 100. Examples of remote data sources include the output of the network scanning tools, the output of a service level agreement server that continuously loads a web page provided by a software host, assuming that software host 100 is a web server, information from software update customer service representatives, and information from software help desk ticketing systems.
Other examples of local performance data sources include Microsoft Event Viewer Log and Task Manager. For UNIX- and LINUX-based systems, SYSLOG information logs may be used as performance data source 116. Another example of a Microsoft-specific performance data source is the NETSTAT command line interface. In yet another example, software update 102 may include its own built in performance monitoring code that monitors the effects of installation of software update 102 on software host 100. In yet another example, the application or update being installed may include a plug in for monitoring performance effects of the update on the software host.
Specific examples of remote performance data sources that may be used include output from network port scanners that indicate the availability of communication ports, information from a customer service representative, and help desk ticketing system information.
In general, performance data that is monitored may include operational and business function data that can be used to calculate a score. Some examples of performance data include CPU usage of software host 100, network connectivity between software host 100 and a target host, the number of help desk tickets open for software host 100, the availability of an application, the availability of network ports. The following examples are examples of system commands that may be executed on software host 100 and the corresponding performance data:
The “NETSTAT” command output shows connections:
The “NET START” command output shows what applications are running: These Windows services are started:
The first example above is the output of a NETSTAT command on a Windows system. In the illustrated example, the NETSTAT command indicates that three TCP applications are listening on three different ports of a device and that there is an active SSL connection with another device. Such output may be used to evaluate the communication availability impact of software update 102. In the second performance example above, the output of the NET START command may be used to indicate what applications are available on software host 100.
Performance monitor 112 may monitor any performance-related parameters 114 and generate corresponding performance values. Examples of actions that may be performed by performance monitor 112 in generating the parameter values include:
The above-listed examples are intended to be illustrative of actions that may be performed by performance monitor 112 in generating the parameter values. However, performance monitor 112 is not limited to these specific actions. Any actions for collecting values that indicate the effect of software update 102 on software host 100 are intended to be within the scope of the subject matter described herein.
Returning to
In one implementation, a score may be a representation of post-installation performance of software host 100 in relation to software update 102. An individual score may represent a post-installation performance of a single software host, such as software host 100. An aggregated score may be representative of post-installation performance of multiple software hosts. The score value may be dependent upon a number of metrics used in score criteria 122A. For example, the score may be a number on a scale 1 to 10.
As stated above, score criteria 122A or 122B may include metrics or tolerances for results collected by performance monitor 112. Each measurement may also include a point value that is used to compute the score. The following table illustrates an example of individual score criteria that may be used to generate an individual score relating to the performance of effects of software update 102 on software host 100.
In Table 1, the left-hand column lists performance rules or criteria to which the performance data values collected by performance monitor 112 are compared. The right-hand column lists corresponding point values for each rule. In the illustrated example, performance rules for CPU usage, port availability, and number of help desk tickets are illustrated. Measured performance data values may be compared to the rules in Table 1 and corresponding points are generated. The points for each performance data value may be then combined to generate a total point score indicative of the overall performance effect of the installation of software update 102 on host 100. For example, if the CPU usage is 20%, the availability of port 80 is 90%, and no help desk tickets are generated when a software update is installed on a software host, the score may be 3+3+3=9.
As discussed above, another aspect of the subject matter described herein may include aggregating performance scores from installations of software update 102 on multiple difference software hosts. Such aggregation may be performed by score generator 118B comparing individual scores and information regarding the diversity of software host test configurations to aggregated score criteria 124. Table 2 shown below illustrates an example of aggregated score criteria that may be used to generate an aggregated score.
In Table 2, the left-hand column includes aggregated score criteria or rules to which individual scores and software host test configuration information are compared. The right-hand column includes corresponding point values. The point values may be combined to generate a total aggregate score for a software update. In one example, a software update may be tested on 3 different CPUs on 3 different hardware platforms with 2 different operating systems. The average individual score for these tests may be 7. Using the data in Table 2, the aggregate score will be 2+2+1+2=7. Because an aggregate score can be based on performance effects of a software update on multiple different software hosts, the aggregate score may provide a universal or platform-neutral indication of the effect of a software update on a software host. The aggregate score and/or the individual score may be used in performing an action with regard to the software update, such as maintaining the software update on a software host in response to the score being greater than a threshold, removing the software update from the software host in response to the score being less than the threshold, or installing the software update on like devices in response to the score being greater than the threshold. Another example of an action that may be performed when the score exceeds a threshold value is to allow installation of the next software update in a sequence of software updates.
Returning to
In Table 3, the left-most field in the update event record includes an identifier for the record. The second field includes a software host identifier, which identifies the software host. In the illustrated example, the software host identifier is an IP address. This identifier may be used to locate configuration information for the software host in another table that stores the corresponding configuration information, such as processor speed, operating system version, other applications being executed, and/or vendor/manufacturer/model number information. The next field in Table 3 identifies the software update. The next field identifies the individual score generated with regard to the software update. The right-most field indicates the performance data that was monitored.
As stated above, actions that may be performed relating to the individual or aggregate score may include removing software update 102 from software host 100 if the score is below a predetermined threshold value or maintaining software update 102 on software host 100 if the score is greater than or equal to the threshold value. Such action may be performed automatically by vetting score client 110 or by update provider 108. Another action that be performed based on the score may include installing software update 102 on devices like software host 100 if the score is above the threshold. This action may be automatically performed by update provider 108. Alternatively, the installation, maintenance, or removal of software update 102 based on the score may be performed manually by a user.
In block 202, different parameters indicative of effects of the software update on the software host are monitored and corresponding parameter values are obtained. Examples of performance parameters monitored include any of the parameters discussed above. In block 204, a score is determined based on the parameter values. The score is indicative of the overall performance effect of the software update on the software host. In block 206, an action relating to the software update is performed based on the score. As stated above, actions may include communicating the score to service provider 104, maintaining software update 102 on software host 100, removing software update 102 from software host 100, or installing software update 102 on like software hosts.
Once the install is complete, control proceeds to block 308 where vetting score client 110 collects information about the software update. Exemplary information that may be collected includes identification information regarding the software update and configuration information regarding the software host. In block 310, vetting score client 110 may generate an update event record for storing performance data, such as the performance parameters monitored, the parameter values, and the score, relating to the software host. In block 312, vetting score client 110 initiates performance monitor 112 to monitor performance-related parameters regarding the effect of software update on software host 100 and initiates execution of the software update.
Referring to
In block 324, vetting score client 110 stores the score in the update event record. In block 326, vetting score client 110 communicates the update event record to update event manager 126.
In block 404, service provider 104 performs an action based on the score. Performing an action may include generating the update event record, computing an aggregate score, installing the software update on devices similar to software host 100, maintaining the software update or software host 100, or removing the software update from software host 100.
Once the installation is complete, control proceeds to block 508 where update event manager 126 collects information about the software update and/or software host 100. Such information may include identification information regarding the software update and configuration information regarding software host 100. In block 510, update event manager 126 generates an update event record. In block 512, update event manager 126 initiates communication with vetting score client 110.
Referring to
In block 520, score calculator 120B processes the score data. In block 522, score calculator 120B compares the performance data to score criteria 122B to generate an individual score relating to the performance effect of installing software update 102 on software host 100. In block 524, the computed score is provided to update event manager 126. In block 526, update event manager 126 stores the score in the update event record.
As stated above, the components for generating the score and collecting performance data may be located locally or remotely with regard to software host 100.
According to one aspect, the subject matter described herein includes a system for obtaining and utilizing a score indicative of an overall performance effect of a software update on a software host. The system may include means for applying a software update for updating a software host. For example, update provider 108 illustrated in
In an alternate implementation from an update or service provider's perspective, a system for obtaining and utilizing a score indicative of an overall performance effect of a software update on a software host may include means for providing a software update for updating a software host. For example, update provider 108 illustrated in
It will be understood that various details of the invention may be changed without departing from the scope of the invention. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation.
