Patent Application
20140026002
The present invention relates in general to methods for utilizing software that interacts with hardware for reporting errors encountered by the hardware. Embodiments of the present invention are particularly well suited, but in no way limited to, software for reporting causal relationships between errors in components and software for determining root cause errors in software controlled electrical or mechanical systems, such as diagnostic test equipment and instruments.
In software systems that interact with real-world hardware components, errors are often difficult to trace to a root cause. An error encountered by software interacting with one hardware component may cause a number of subsequent follow-on errors in other components or software modules interacting with these hardware components. In a software/hardware system that is capable of recording or displaying errors, multiple errors may be generated across a variety of internal subsystems in response to a root problem with the hardware or software module that encountered a problem. Displaying these multiple related errors and associating them in a meaningful way can be a difficult problem.
In order for an operator or service engineer to successfully diagnose and resolve the overall problem in the system, the operator needs to determine the originating root cause error and treat it. The follow-on errors that may be generated from a root cause error are generally not important when diagnosing and treating the overall problem. However, in many prior art systems, the resulting errors are displayed to the operator, regardless of whether they are a likely root cause or not, in an unarranged, undistinguished fashion alongside the original root cause error. This problem may even exist where the systems have a means for associating errors.
There is often no simple way for the operator or engineer to distinguish which error is the root cause and which errors are follow-on errors (e.g., those that are often less important from the stand point of solving the problem—merely symptoms that result from the root cause error). The operator often needs to guess the root cause error based on other information such as error timing, experience, complex log files, or luck. This guesswork can be time-consuming and costly for the operator and/or service engineer and is often detrimental to the reliability of the hardware/software system.
In some systems, certain errors may be considered more critical than others. For example, an error that results in a total system shutdown and significant loss of productivity may be considered more severe than a simple timing error that results in a minimal loss of productivity. However, in many prior art systems, the overall severity of the root cause error may not be known until the follow-on errors are generated and manually correlated by the operator back to the root cause error. Particularly in systems where errors of lesser initial severity can be automatically hidden from the operator, it may be difficult to determine that an error that was hidden due to low severity is actually a root cause error of significance if it is not associated with more severe follow-on errors. For example, it may not be possible for the instrument to display a particular root cause error to the operator if the system does not determine the error is severe enough. Many prior art systems typically lack knowledge about the causal connections within a group of errors and this frequently prevents the true severity of the originating error from being known.
There is a need for a system that automatically determines the causality between a group of errors. Such a system that reliably solves this problem has previously been considered extremely difficult if not impossible. While conventional systems can easily determine causal links in some cases, other cases were considered too difficult or had no known solution. In order for error-causality systems to be useful, they must be able to determine the common causalities that occur. This may be difficult if the software in the system has not been designed with this goal in mind. In particular, establishing the causality of software thread time-out errors (e.g., a software thread timing out on a lock held by another thread that is processing a different error) has been a difficult problem in system design.
An example of a system where this need exists can be found in U.S. Pat. No. 7,381,370, assigned to the assignee of the present application and incorporated herein by reference. In complicated instruments, such as chemical analyzers, which may include a number of precisely moving parts, a root cause error may be simple in the real world but difficult to determine in software. For example, if a servo motor has become worn or stuck, it may result in errors in other mechanical portions of the instrument that interact with the motor. There is a specific need in chemical analyzers, medical devices, or other complex software/mechanical instruments to provide software mechanisms that simplify, repair, and diagnose when errors occur in the system.
The present invention provides software mechanisms that facilitate the association of errors to assist in the determination of root cause errors versus follow-on errors. The present invention also provides software mechanisms to facilitate simplified displaying of errors to assist operators in determining root cause errors.
Embodiments of the present invention address and overcome the above shortcomings and drawbacks by providing improved methods for identifying parent-child relationships among errors. By associating errors by a causal relationship, errors can be displayed as graphs or trees. In addition, improved methods of assigning severity levels to errors within the graph allows for simplified filtering of less severe errors, without masking likely root cause errors. This technology is particularly well-suited for, but by no means limited to, software systems for interacting with hardware systems, such as diagnostic test equipment and instrumentation.
Embodiments of the present invention are directed to a method for reporting errors in a software environment including a first step of generating a first error by a first software thread, including a first error ID, in response to a first error condition relating to the state of at least one resource. The method further includes propagating the first error ID to at least one second software thread, generating a second error by the second software thread, including a second error ID and the first error ID, in response to a second error condition relating to the state of at least one resource, and reporting the first and second errors to a user.
According to one aspect of the invention, the method for reporting errors further includes the step of reporting the errors to a user interface. According to another aspect of the invention the first and second IDs comprise unique identifiers. According to a further aspect of the invention, the method for reporting errors further includes propagating the first error ID to at least one third software thread, generating a third error by the third software thread, including a third error ID and at least one of the first or second error IDs, and reporting the third error to the user.
According to one aspect of the invention, the step of propagating the first error ID includes updating an attribute in a software object that is accessible to both the first and second threads. According to another aspect of the invention, a software object includes a lock, such that only one of the first and second threads can control the resource at any given time.
Embodiments of the present invention are directed to a method for reporting errors in a software environment, including requesting, by a first software thread, access to a shared resource guarded by a software object controlled by a second software thread. The method further includes determining whether a time-out condition has occurred while the first thread is awaiting access to the shared resource, generating an error in response to the determining step, including a first error that includes a first error ID, obtaining from the software object a second error ID associated with the second thread, and associating the first and second error IDs. The method can then report the first error along with an association between the first and second error IDs to a user.
According to one aspect of the invention, reporting errors includes reporting the errors to a user interface. According to another aspect of the invention, the first and second IDs are unique identifiers. According to a further aspect of the invention, the method for reporting errors further includes updating information in the software object, by the second thread, to include the second error ID when the second thread encounters an error while the second thread has control over the software object.
According to one aspect of the invention, the software object is a lock, such that only one of the first and second threads can control the hardware resource at any given time. According to another aspect of the invention, the software object is an object that propagates a signal from the first thread to the second thread upon completion of an instruction sequence by the first thread. According to an additional aspect of the invention, the shared resource is a shared hardware and/or shared software resource.
Embodiments of the present invention are directed to a method for displaying errors to a user, including gathering a plurality of errors representing hardware or software error conditions. The method further includes associating the plurality of errors into a graph to convey causal relationships amongst the errors, including at least one parent error and one or more child errors and displaying the graph of errors to a user. Causal relationships can be determined based on identifiers within the errors.
According to one aspect of the invention, each error has a predetermined severity. According to another aspect of the invention, the method for displaying errors to a user further includes determining a severity of each of the plurality of errors and displaying those errors that have a severity above a threshold. According to a further aspect of the invention, at least one parent error is displayed where at least one of the one or more child errors exceeds the threshold. According to yet another aspect of the invention, a parent error can be assigned the highest level of severity determined for the one or more child errors. According to a further aspect of the invention, one or more child errors are masked when an assigned severity of a parent exceeds an assigned priority of the child error.
Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments that proceeds with reference to the accompanying drawings.
The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:
The above problems and shortcomings in the prior art have motivated the creation of methods for identifying the causal relationships (described herein as “parent-child” relationships) between errors encountered by software or hardware in a system that includes multi-threaded software and shared resources. Exemplary resources can include sensors, electromechanical components or subsystems, or any other resource that may be shared amongst threads. Threads pass information about errors encountered to the shared resources or to other threads so that subsequent errors can associate the passed information with the cause for subsequent errors.
An instrument, such as chemical analyzer, includes software that interacts with, and controls, mechanical components. An exemplary system of a chemical analyzer that uses software control is described in detail below. An exemplary instrument containing a software/mechanical system can include the Dimension® clinical chemistry analyzer sold by Siemens Healthcare Diagnostics Inc. of Deerfield, Ill.
An exemplary software/mechanical system can include multiple threads that interact with software objects. These objects can model or control hardware components in the system. For example, a motor in the system may be represented by one or more software objects that can be controlled or interacted with by the threads in the system. The software object may include drivers or other control mechanisms for controlling a hardware component; the object may further include data representing the real-world state of the component or sensor information. Within the system, multiple threads may need to interact with the hardware component and, more particularly, the software object representing the hardware component. These threads may include, for example, different control routines within the system. For example, a thread may include the routines necessary for a procedure that uses multiple components and hardware/software systems, such as a homing routine. These may include startup and diagnostic routines or routines that run frequently in the system. Other threads can include:
Objects or threads can include error handling routines. When a thread encounters an error, it can report this error via these handling routines. Error handling routines can include sending a notice of the error to a log file, display, or any other useful repository of error information and may include predetermined routines such as returning the system to a known state or attempting the routine that encountered the error again. Error handling can include creating one or more objects that represent the error. They can also include modifying attributes of an existing object representing one or more errors. By utilizing an object oriented approach, some embodiments are able to more easily associate multiple errors with a common cause and propagate information between related errors.
An error can include a problem encountered by hardware or software. An exemplary type of error can include determining that a hardware or software state differs from an expected state. In some embodiments, errors may be represented using a common format, such as a structure or object that allows the system to handle hardware and software errors in similar fashion. In some embodiments, errors encountered by hardware can be treated separately from errors encountered by software. Parent child relationships can be created between hardware and/or software errors. It should be noted that in many systems in which the present invention can be employed, hardware errors can cause software errors, and vice versa. Accordingly, it will be appreciated that the decision to treat hardware and software errors the same or separately can depend on the type of system employing the present invention.
One common method in the prior art for reporting errors includes storing error codes associated with an error and may include storing a timestamp indicating the system time when the error occurred. In systems where error reporting may be delayed, or where an operator reviews errors, a timestamp may be useful for associating errors. However, relying on timestamps can be misleading. For example, where an error is caused by interaction with a real-world device, related errors may not be noted by the software at the same time and related errors may have different timestamps. Complicating the matter, the root cause error may not be the error with the earliest timestamp. Timestamps, however, may still be useful to the operator and can be displayed in some embodiments of the present invention.
Errors often result in a related chain of errors. To facilitate association of errors within the same chain, some embodiments utilize an error model where each error includes multiple attributes. A first attribute can include a unique serial number associated with this particular error. A second attribute can include the serial number that identifies a parent error. A parent error, as used herein, is an error that is determined to be the cause of the present error. Likewise, a child error is an error that is determined to be caused by the present error. For example, a parent error might include a motor encoding error, while a child error may include the resulting positional encoding error. For ease of reference, the serial number of the present error can be referred to as the “instance ID,” while the serial number of the parent error can be referred to as the “parent instance ID.” It should be appreciated that the parent instance ID can include a serial number and could additionally or alternatively include a pointer to a parent error.
In some embodiments, each new error is handled by creating a new error object including instance ID and parent instance ID attributes. Other attributes may include, for example, the type of error, the timestamp of the error, the severity of the error, the thread ID that instantiated the error, the hardware or software associated with the error, or any other information that may be useful for reporting the error to an operator of the system.
There are a few general types of error situations that may be encountered by embodiments of the present invention. A first situation 100, shown in
A second situation 200 that may be encountered is a “top down” error, shown in
Multiple threads (e.g., 220, 222, and 224) that are operating within the system may create “thread interrupted” errors (e.g., 225, 227, and 229) as part of an error handling routine. In this scenario, without associating the “halt instrument” error 215 as a parent error to the resulting errors, such as “thread interrupted” errors 225, 227, and 229, it may become difficult to later determine a root cause error (e.g., the “halt instrument” error 215). To solve this, when creating the “thread interrupted” error 225, the thread 220 may assign attributes that include the parent instance ID of the “halt instrument” error 215 and a new instance ID for the current “thread interrupted” error 225. This allows the system to associate many errors easily with a single root cause parent error.
This concept is shown in
Including the parent ID for a “top-down” error can be further facilitated by including an instance ID associated with the source of the interrupt signal in the interrupt signal. For example, when the interrupt is represented by an interrupt object 210, the interrupt object 210 may include an attribute that identifies the instance ID of the error 215 associated with the creation of the interrupt object 210. As a result, thread interrupt code (e.g., routines in threads 220, 222, and 224) that watches for interrupt signals can extract the instance ID of the error that caused the interrupt. The error handling routine responding to the interrupt can then include the instance ID (recovered from the interrupt signal) as the parent instance ID of a resulting “thread interrupted” error.
A third situation includes time-out errors. Time-out errors have traditionally been among the most difficult to associate for determining the root cause. An exemplary time-out error might include a “locks not cleared” error. For example, a first thread, thread A, may have control of a software lock associated with a hardware device or other resource, such as a memory device. A second thread, thread B, may be waiting to use this resource and may be waiting to acquire control of the software lock associated with the resource or device. If thread A encounters an error while controlling the lock and consequently does not release the lock in a timely manner, then thread B may be left waiting for thread A to release control of the software lock for the hardware device. Thread B may encounter a time-out condition. As a result, thread B may generate an error to avoid idling indefinitely while waiting for the lock to become available. Accordingly, thread A and thread B may both generate errors that occur at different system times. Traditionally, there has been no obvious way to associate these two errors. Embodiments of the present invention provide a method for associating errors generated by multiple threads having a causal relationship. This can enable the creation of a parent-child relationship between the error generated by thread A and the error generated by thread B. This can facilitate the determination of a root cause for errors occurring in multiple threads due to timing errors.
This concept can be illustrated with respect to
The middle column in
While thread A has control of resource 260 and the lock of resource 260 is in a locked state 264, another thread, such as thread B, may seek to acquire a lock and permission to use resource 260. For example, where thread B seeks to perform a motion task using the robot arm associated with resource 260, thread B may notify resource 260 of its intent to use the resource exclusively by sending an “acquire” message 272. Resource 260 may notify thread B that it is currently unavailable (such as by not responding to the acquire message 272 until resource 260 has been locked for use by thread B or by sending a “wait” instruction).
Thread A releases resource 260 when it is finished using the resource. In some embodiments, thread A actively releases the lock state 264 on resource 260 by sending a “release” message 254. It should be noted that in some embodiments, resource 260 could perform operations at the direction of thread A and release the lock 264 automatically upon completion by changing the state of the lock and sending a release notification (e.g., task complete message) to thread A. If other threads are waiting to lock or use resource 260, resource 260 can determine which threads have pending acquire requests, such as request 272, and assign control to the chosen waiting thread. During period 266, the lock is assigned to thread B, such that thread B is no longer waiting and can perform whatever tasks it was waiting to perform. Upon completion, thread B releases the lock from the locked state 266 in the same manner as thread A, such as via a “release” instruction 274. Resource 260 then returns to an unlocked state (e.g., 268) where there are no other threads waiting for control.
In this example, thread A encounters an error condition while it has control of the lock of resource 260, causing thread A to create an error 255. This error may include a parent instance ID if caused by another error or event, and includes a unique instance ID (e.g., 255). In prior art systems, thread A and thread B might have no way of knowing about one another if they do not directly interact. While they compete for a common resource, they may not have any paths of communication. Without some manner of passing error information to other threads that may be waiting to use resource 260, resulting time-out errors may not easily be associated with error 255, which may be the root cause (or at least a cause) of these subsequent errors.
To facilitate an association between error 255 and any resulting time-out errors generated in other waiting threads, thread A passes information to the lock in resource 260. Where the lock is an object, thread A may simply set an attribute dedicated to propagating error information. Upon encountering (or creating) an error 255, thread A may set an ID value within resource 260 (or a lock object associated with resource 260) to include information about error 255, such as the error's instance ID (255). In some embodiments, the information passed to the lock may include any parent instance IDs associated with error 255.
Upon encountering a time-out condition, thread B may create an error 275 to report this condition. In prior art systems, thread B may not have known of any parent errors that caused the time-out error. As a result, time-out errors would not be associated easily with the other errors that caused the time-out. In the scenario shown in
If thread B controls any locks that cannot be released, thread B can propagate its error information in the same way as explained for thread A. For example, if thread B had acquired a lock to another resource while waiting for resource 260, thread B can update an error ID attribute in that other resource so that threads waiting for that resource can subsequently include a parent relationship to error 275. In some embodiments, thread B will propagate the instance ID of error 275 to the resources it has locked. In these embodiments, the resulting error tree will be multi-level, with error 255 as a grandparent of errors created by any threads waiting on resources locked by thread B. In some embodiments, thread B can additionally or alternatively propagate the parent instance ID (255) to those held resources. In these embodiments, the resulting error graph will be a flatter tree with subsequent errors appearing as children of error 255, rather than error 275. In either scenario, error 255 will be deduced as a cause of all the resulting errors. Error 255 can be considered a potential root cause where it has no associated parent errors.
Eventually, thread A may be able to recover from error 255 and complete its exclusive use of resource 260. For example, where thread A experienced a time-out error on another resource needed to complete the use of resource 260, thread A may have subsequently acquired that other resource, enabling thread A to complete its task. Upon completion of thread A's use of resource 260, the resource moves into an unlocked state. At that point, the error ID (255) can be cleared, as error 255 should no longer cause an unresolved lock of resource 260. This prevents subsequent threads from discovering a stale error ID and misinterpreting error 255 as a cause of the subsequent error.
In some embodiments, threads acquiring a lock to resource 260 can place their thread ID information in the lock object. This can allow subsequent time-out errors that may be encountered by other threads to include the information about the thread that controlled the lock at the time of the time-out. This information can be useful in determining what caused the error where thread A did not experience an error 255, but thread B nonetheless encountered an error 275. In this example where error 275 is considered a root cause error candidate, the information about the state of the lock to resource 260 can be propagated as part of an error reporting scheme, and can be useful in determining what caused the error 275.
At step 400, thread A enters a new application dependent sequence. At step 402, the application dependent sequence is determined. Particularly, step 402 determines what type of application dependent sequence must be executed. This determination may result in a recursive branch to an application dependent sequence 400. This determination may also result in the execution of application dependent instructions 404 that are defined in the code being executed. The application dependent instructions executed at step 404 can include any number of instructions chosen to carry out a task by the thread. The determination at step 402 can also result in the execution of a signalable sequence 406, execution of a waitable sequence 408, or execution of a lockable sequence 410. Each of these sequences can make further recursive calls to application dependent sequences 400. Upon completion of the execution step, the application dependent sequence determines if an error has been returned by an execution step, at step 412. If an error has been returned, the sequence will execute error handling 414. Upon completion of error determination and error handling, the application dependent sequence will determine if further execution is needed at step 416. If further execution is needed, the sequence repeats beginning with determining the application dependent sequence to be executed at step 402. Otherwise, the application dependent sequence finishes and returns.
The various sequences that can be executed depend on the type of object being interacted with by the application dependent sequence 400. There are two primary types of objects that are of interest to the sequence shown in
Another type of waitable object is a lockable object, such as resource 260 (
Threads can maintain a registry or list of all waitable objects and/or threads that should be alerted in the event of an error. For simplicity, this registry will be referred to herein as the intent registry. The intent registry can be used by threads to complete a signalable sequence or a lockable sequence, or to track resources that should be used to propagate errors to other threads that may be affected by the completion of a signalable or lockable sequence. For example, in the case where thread A encounters an error while holding a resource, thread A sets an error instance ID on the resource. Thread A can determine which objects to set an error instance ID on by examining an intent registry. This can be done even when no other threads are currently waiting on the resource. When thread A releases the resource, it clears the error instance ID. In the case where there are no other threads waiting on the resources, thread A's error instance ID will not affect any other thread and has no effect on other errors because no other thread uses the resource during this time period. Setting and clearing the ID on a resource when an error occurs is an unobtrusive action when no other threads wait on the resource. Setting the ID even when no threads wait on the resource can be a proactive measure for the case that one or more other threads end up waiting on the resource in the future.
The intent registry can be separate for each thread and maintained by each thread, or may include a common registry that is updated by actions of the plurality of threads that use the common intent registry. In some embodiments, the intent registry can include a supervisor thread that facilitates the maintenance of the intent registry. Actions using the intent registry can include adding an association of a thread's intent and a target object (e.g., the intent to signal a signalable object or the intent to lock/unlock a lockable object); unregistering the association (e.g., removing the intent to signal/lock/unlock, such as after completion of the intended action); or querying the intent registry for associations, such as during error reporting or diagnostics to determine error causality.
The intent registry can be implemented in any appropriate form for the application, such as a list, linked list, array or vector, database, or the like, and can reference objects via IDs or pointers. For example, the intent list can include a common database of waitable objects referenced and searchable by an associated thread ID. In some embodiments, the intent list can include a storage list of waitable objects that is local to each thread. In some embodiments, the intent registry can include uphill and downhill relationships between threads and waitable objects, meaning, the intent registry can include the waitable objects that are waiting to be signaled or unlocked by the thread, as well as the waitable objects that the thread is waiting on to become available or to send a signal to the thread. Accordingly, the intent registry can be used to determine which objects (or threads) thread A intends to signal/lock/unlock, or which threads intend to signal/lock/unlock a given resource. The intent registry can include one-to-one, one-to-many, and/or many-to-one relationships, as appropriate.
Signalable sequence 406 is shown in
Execution begins at step 420, where thread A identifies the object to be signaled and registers its intent to signal the signalable object in the intent registry. This enables thread A to propagate any error encountered to objects that may be used by peer threads where the failure to timely signal the object may cause time-out errors in peer threads. The intent registry can also be used to identify the signalable object or objects that should be signaled upon completion on the signalable sequence.
Registration at step 420 may include sending information to the lockable object or may include recording (e.g., by ID or by pointer) the lockable object in a list of objects that the thread intends to signal (e.g., the intent registry and/or other data structures). By adding the objects that are expecting a signal as part of the execution of the signalable sequence, thread A can identify those signalable objects that should receive a responsive signal upon completion of execution. By maintaining the intent registry, thread A knows which objects may be relied on by other threads in the event of an error. The intent registry can be useful for propagating error information to the other threads (e.g., thread B) that are expecting a signal should an error prevent thread A from sending the signal. In embodiments where the signalable object may be exclusively controlled, the step of registering intent to signal the object can include clearing any error ID that is stored in the signalable object. For example, these prior error IDs are likely the result of past errors encountered by thread A, and are likely stale. This process is similar to state 268 shown in
Once the signalable object is registered at step 420, the application dependent sequence 400 is then executed. Upon completion of the application dependent sequence, the signalable sequence determines if the application dependent sequence 400 resulted in an error, at step 422. This step can be via any conventional error detection means such as exception handling, identifying an error object created, or receiving reference to the error upon return of the application dependent sequence.
If an error is detected, execution proceeds to step 424 where the thread will examine the intent registry and can use the registered objects to propagate error information. At step 424 thread A will post the instance ID of the error (e.g., 255,
If, on the other hand, no error is encountered by the application dependent sequence 400, the signalable sequence proceeds to step 426 where the expected signal is sent to the signalable object. This signal may include any expected signal such as a result, object, or a confirmation. As a result of this step, any other threads that are waiting for the signal via the signalable object can complete their waitable sequence. After signaling the object, the signalable sequence proceeds to step 428 where the thread unregisters the signalable object from the intent registry. This step may include sending information to the signalable object and removing the signalable object from the list of objects in the intent registry used by thread A (or a global intent registry if used).
As a further illustration of the ability of threads to share resources and propagate and associate error information, pseudo-code may be considered. The following pseudo-code is a representation of a shared locking resource between threads where one thread currently has access to the resource (such as thread A and shared resource 260 in
The following pseudo-code illustrates the representation provided by
The following pseudo-code illustrates the representation provided by
Similarly, the following pseudo-code is provided to reflect a signalable object (i.e., an object that allows one thread to pass information to another thread):
The following pseudo-code illustrates the representation in which an error instance ID is created upon a thread encountering an error when accessing a signalable object:
The following pseudo-code illustrates the representation in which an error instance ID is created and also maintained in an intent registry upon a thread encountering an error when accessing a signalable object:
Lockable sequence 410 is executed by thread A when interacting with a lockable object (e.g., resource 260,
At step 442, thread A determines whether it has been granted exclusive access to the lockable object or whether a time-out error has occurred A time-out error may occur if the thread has not been granted access within a predetermined time period (e.g., error 275,
If thread A has been granted access to the lockable object without encountering a time-out error, at step 442, thread A will then acquire the control of the lockable object. Acquisition can be by any suitable method such as changing an ownership attribute in the object, registering an active relationship between thread A and the object, setting a semaphore, or the like. At step 450, the thread will register its intent to unlock the object with the intent registry to identify the lockable object as locked by thread A. This registry may be the same as the intent registry discussed above with respect to signalable objects, or may be maintained as a separate unlock intent registry (for ease of reference, this list will be referred to as a common intent registry). As described above, the intent registry includes those objects that should be notified by thread A if an error occurs. In addition, thread A may clear any previously entered error IDs from the lockable object to prevent any other threads that are waiting for a lockable object from processing subsequent time-out errors using stale parent error information. In some embodiments, at step 450, thread A may set an ownership attribute in the lockable object to the thread ID of thread A so that any potential time-out errors can be traced to a time-out that occurred while thread A had control of the lockable object, even where thread A did not encounter an error. This information may be useful for determining the conditions encountered by a subsequent thread if a time-out error is identified as the root cause error.
After acquiring a lockable object, thread A can then execute any application dependent sequence necessary to perform the lockable sequence. This can invoke another application dependent sequence to carry out any routines that should be executed while thread A has control of the lockable object. At step 452, thread A determines whether or not the application dependent sequence 400 resulted in any errors. If so, at step 454 thread A will propagate information about error (e.g., 255,
At step 456, if no error was encountered during the application dependent sequence, thread A may unregister (e.g., remove) the lockable object from the thread's intent registry. This will prevent thread A from propagating an error ID to lockable objects that are no longer locked. At step 458, the thread will release the lockable object, such as by sending a release signal 254 (
At step 462, the error handling routine 414 determines the type of error handling. For example, the error handling routines may be set up to execute an application dependent sequence 400 containing predetermined instructions. The error handling routines may also be set up to send the error to the user interface or a log file or otherwise report the error so that such an error can be later reviewed, at step 464. The reporting can include attributes of the error such as type, time stamp (if available), severity, instance ID, parent ID, and any other attributes determined to be useful. The treatment of these reported errors will be described in detail below.
When the error handling routine relates to a signalable sequence, error handling proceeds to step 466. At step 466, thread A can perform a cleanup process, such as placing a signalable object in a predetermined state for safe error handling. Once the signalable object is cleaned up, at step 470, thread A will unregister its intent to signal the signalable object in the intent registry, as the objects may no longer need to be signaled after the error. The thread may also clear any error IDs in this signal object.
When the error handling routine relates to a lockable sequence, error handling proceeds to step 468. At step 468, thread A can perform a cleanup process such as placing a lockable object in a predetermined state. Once the error has been properly handled, it may be safe to release the lock on the resource. This determination can vary by the type of error encountered. If it is safe to unlock the lockable object, thread A will unregister its intent to signal the signalable object in the intent registry at step 480. Thread A can then release the lockable object at step 482 such that another thread can control the lockable object and, by extension, the associated resource.
At step 490, the error handling routine determines if any other error handling steps are needed to complete the handling of the error. If not, the error handling routine exits.
As the system software encounters errors, these errors can be reported to the user via a user interface. This user interface can include real-time descriptions of errors, or may include storage and viewing of a log file. An exemplary user interface 500 is shown in
Errors shown in the example in
In the illustrated embodiment depicted in
As can be seen in screen 520, in tree form, an operator can tell at a glance that error 502 is likely the root cause of all the errors on the screen. Even an inexperienced operator can easily make this determination.
In addition, column 525 may include severity information to indicate the severity of the errors encountered. User interface screen 520 can include filter preferences that enable a user to mask less-severe errors. The severities of these errors can be conveyed in any reasonable manner, such as a numeric value, a range, a shape, a symbol, color, motion, size, brightness, etc. It is worth noting that in exemplary screen 520, errors 504, which are children of error 502, have been displayed with a higher level of severity than root cause error 502. It is important to note that the root cause error may not be considered by the system automatically to be the most severe error. Therefore, as explained in detail below, simply filtering the display of errors based on an initial severity determination may not be sufficient to inform the operator of a root cause error.
A more complex example of errors that can be displayed via a user interface is shown in
By switching to a tree view via button 610, a user may display a graph of the parent-child relationships to better understand what happened. As shown by the user interface screen 620 in
When introducing the concept of severity to these errors, it may frustrate the creation of the graph 700. For example, if we apply severity levels described below to the example in
To solve this problem, some embodiments propagate children severities up to parents once a graph of errors is compiled. These embodiments can use a rule (explained in
This concept is illustrated in
Once error severity levels are propagated, the display can apply filters to display important errors and their parents, while masking less severe errors to reduce the noise seen by an operator. This can simplify the error display while maintaining the most important information. For example, a filter could display only those errors above a threshold severity, such as level 4. When higher severities of the children are propagated to parent errors, a simple threshold filter can produce pruned trees that include the root cause error. Alternatively, an available filter may also include logic that allows children errors to be masked when less severe than the parent error. Applying such a filter allows the user to see the most severe errors in the tree and easily trace the causality of the most severe resulting errors.
For example,
The user can be given the option to display the full error graphs for more information. This can be useful for showing all the effects from root errors, or showing any other errors that were suppressed, such as a motor error that only caused a test to quietly rerun. If the user is a non-engineer operator that does not normally care about what low-level errors caused the instrument to halt, the user can select a filtered view, such as shown in
Examples of the meanings of severity levels are shown in Table 1. Examples of errors and their default severity levels are shown in Table 2. In some embodiments, the meanings of severity levels and the associations between severity levels and errors are configurable and customizable.
The methods discussed herein can be used in a system that encounters errors. It is well suited to hardware and software systems that include multi-threaded processes and use locks on shared resources. An exemplary system that can benefit from these techniques is the chemical analyzer shown in
Reaction carousel 12 is rotatable using stepwise movements in a constant direction, the stepwise movements being separated by a constant dwell time during which reaction carousel 12 is maintained stationary and computer controlled assay operational devices 13, such as sensors, reagent add stations, mixing stations, and the like, operate as needed on an assay mixture contained within a cuvette.
Analyzer 10 is controlled by software executed by a computer 15 based on computer programs written in a machine language like that used on the Dimension® clinical chemistry analyzer sold by Siemens Healthcare Diagnostics Inc. of Deerfield, Ill., and widely used by those skilled in the art of computer-based electromechanical control programming. Computer 15 also executes application software programs, such as the Dimension Vista® system software for performing assays conducted by various analyzing means (e.g., detection units) 17 within analyzer 10. Analyzing means can include, for instance, one or more photometers, turbidimeters, nephelometers, electrodes, electromagnets, and/or LOCI® readers for interpreting the results of reactions within the reaction vessels or cuvettes.
As seen in
A conventional liquid sampling probe 42 is located proximate the second end of the input lane 35 and is operable to aspirate aliquot portions of sample fluid from sample fluid tubes 40 and to dispense an aliquot portion of the sample fluid into one or more of a plurality of vessels in aliquot vessel array 44. This provides a quantity of sample fluid to facilitate assays and to provide for a sample fluid aliquot to be retained by analyzer 10 within an environmental chamber 48. After sample fluid is aspirated from all sample fluid tubes 40 on a rack 38 and dispensed into aliquot vessels in array 44 and maintained in an aliquot vessel array storage and transport system 50, rack 38 may be moved, as indicated by open arrow 36A, to a front area of analyzer 10 accessible to an operator so that racks 38 may be unloaded from analyzer 10.
Sample aspiration probe 54 is controlled by computer 15 and is adapted to aspirate a controlled amount of sample from individual aliquot vessels in array 44 positioned at a sampling location within a track (not shown) and is then shuttled to a dispensing location where an appropriate amount of aspirated sample is dispensed into one or more cuvettes for testing by analyzer 10 for one or more analytes. After sample has been dispensed into reaction cuvettes, conventional transfer means move aliquot vessel arrays 44, as required, within aliquot vessel array storage and dispensing module 56 between aliquot vessel array transport system 50, environmental chamber 48, and a disposal area (not shown).
Temperature-controlled storage areas or servers 26, 27, and 28 contain an inventory of multi-compartment elongate reagent cartridges (not shown) loaded into the system via input tray 29, such as those described in U.S. Pat. No. 6,943,030 assigned to the assignee of the present invention, containing reagents in wells 32, as necessary, to perform a number of different assays. Reagents may be moved and aligned within analyzer 10 by any conventional means, including those described in 2009P13675WO, also assigned to the assignee of the present invention, and incorporated herein by reference. Computer 15 can control and track the motion and placement of the reagent cartridges. Reagents from server 27, 28, and 29 can be handled by one or more reagent probe arms, 61, 62.
The methods discussed herein are also suited to be executed by a computer. The computer implemented can be any conventional computing device, such as a PC (such as a desktop computer), server, single or multiprocessor device, handheld device (such as a phone), or device that includes different processors working together across any bus or network (such as a cloud computing environment). These computing devices can be capable of multithreaded or single threaded processing in some embodiments.
It should be noted that root cause may not necessarily be considered an error by the software. A halt button pressed by a user may be handled as an event rather than an error. Embodiments of the invention can still associate subsequent errors with this initial event. For example, events may also be given instance IDs that can be used to associate parent child relationships. It should be understood that the matters discussed herein can be used for associating events, as well as errors.
It should also be noted that a single root cause error may cause one or more child errors and each child error may also include other child errors. Accordingly, by associating child and parent errors, it is possible to create a tree graph that can be used to easily determine root cause errors. It should be noted that in some systems errors may have more than one parent error. Objects presenting these errors can include attributes that identify more than one parent. Accordingly, the resulting graph may be more complicated than a tree. Some embodiments can handle a more complicated graph. However, examples used herein refer to the simple case of a tree for simplicity. These methods should not be considered limited to trees and can be easily applied to multi-parent graphs.
Although the invention has been described with reference to exemplary embodiments, it is not limited thereto. Those skilled in the art will appreciate that numerous changes and modifications may be made to the preferred embodiments of the invention and that such changes and modifications may be made without departing from the true spirit of the invention. It is therefore intended that the appended claims be construed to cover all such equivalent variations as fall within the true spirit and scope of the invention.
This application claims priority to U.S. provisional application Ser. No. 61/473,095 filed Apr. 7, 2011, which is incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US12/32499 | 4/6/2012 | WO | 00 | 10/4/2013 |
| Number | Date | Country | |
|---|---|---|---|
| 61473095 | Apr 2011 | US |
