Patent Application
20090012821
This invention relates to methods and systems for managing one or more live remote medical displays. In particular, a method is disclosed for the management of network bandwidth for network connections between a composite display system workstation and several remote field workstations.
The evolution of medicine, and in particular, of interventional medicine, has led to the development of complex systems integrating patient monitoring sub-systems, imaging systems encompassing several modalities such as x-ray, MRI, ultrasound, nuclear medicine, and optical systems providing various close views of the anatomy being operated on. Additional sub-systems are often required, depending on the specifies of a given medical intervention. As an example, ablation means are often employed in the treatment of various lesions.
Within the field of interventional medicine, the recent trend has been toward the development and deployment of minimally invasive intervention systems include navigation systems, such as the Niobe™ magnetic navigation system developed by Stereotaxis, St. Louis, Mo. Such systems typically comprise an imaging means for real-time guidance and monitoring of the intervention; additional feedback is provided by a three-dimensional (3D) localization system that allows real-time determination of the catheter or interventional device tip position and orientation with respect to the operating room and, through co-registered imaging, with respect to the patient.
As a more specific example of complex interventions only recently made possible, the availability of methods and systems for safe, efficient minimally invasive interventions have greatly impacted and changed the practice of cardiac treatment delivery in the last decade. The treatment of a number of cardiac disorders has become possible without requiring open heart surgery. In particular, minimally invasive vascular and chamber navigation devices and systems have evolved to the point that a great number of heart conditions, such as electrical disorders of the heart rhythm, and arterial-related conditions, such as angina pectoris and heart tissue ischemia, can now be diagnosed and treated through the insertion and navigation of thin elongated devices through an arterial or venous incision and to the region of interest.
As methods and technologies evolve, the range and complexity of conditions amenable to minimally invasive diagnosis and treatment has increased. As a specific example, the field of medical device endo-lumen and endocardiac navigation is greatly specialized, and experts need mastery of disciplines at the cross-road of science, medicine, and engineering to bring about the best outcomes. Due to the risks intrinsic to cardiac surgery and similar procedures, the availability of training and teaching tools, the capability to remotely attend live interventions, and the tools required to allow a remote user to take control of an intervention, are key in the development of the next generation of practitioners and the safe and effective performance of complex operations.
Technology has been developed to enable physicians, call center representatives, and other attendant staff to remotely access multiple live, high resolution displays necessary to monitor and perform medical procedures. Each display and associated controls presents image or other graphic or text data relevant to a sub-system, and a user-interface (UIF) associated with that display enables control of that sub-system. One of the technology evolution challenges is to find a way to remotely view a number of high refresh rate, high resolution medical displays via a network with reasonable bandwidth requirements for broad distribution to medical centers around the world. While many technologies exist today to convert the data stream into data packets, intrinsic network limitation and high information contents provided by a multiplicity of required sub-system components stretch the limits of currently available compression technologies. Other technologies that fail to appropriately compress the video screen include Keyboard-Video-Mouse Transmitter/Receiver Pair (KVM) which attempts to send every pixel via the network causing artifacts limiting the quality and refresh rate, so that live medical information can not be properly interpreted. In addition to display information, keyboard, mouse, and other interfaces or input devices data streams should be transmitted in synchrony with the video or graphic data transmission.
Recently, approaches have been proposed to integrate the output data streams of a collection of information sources associated with a complex medical system, such as a minimally invasive navigation system, and to facilitate single-point command and control of the various sub-system information sources through an integrated, combined workstation system comprising image, graphics, and text display and input and command interface means. For example, patent application Ser. No. 11/484,883 entitled “System and network for remote medical procedures” and included by reference in its entirety describes and claims a system with a central control center with controls for each of a plurality of navigation systems, and patent application Ser. No. 11/670,930 entitled “Global input device for multiple computer-controlled medical systems,” included by reference in its entirety, describes and claims a composite display with seamless cursor movement. The bandwidth necessary for data transfer from each of the controlled sub-systems to an integrated composite display and control workstation is generally not a limitation, as typically at least one such composite workstation will be located in the vicinity of the complex system and associated sub-systems. However practical and economical means required achieving the objectives described above for training, teaching, proctoring, and other such related objectives, are not currently available.
The present invention relates to solutions for network bandwidth management in the context of integrated composite display and command workstation. This invention has been developed to enable physicians, call center representatives, and other attendant staff to remotely access multiple live, high resolution displays necessary to monitor and perform medical procedures. The elements of this invention include multiple systems in a lab each with their own display, a consolidated display of the medical systems, at least one converter to translate the local high resolution display into data for transmission over a network, a network link to transmit the data and at least one receiver to compile the data transmitted over the network into a similar high refresh, high resolution display. By consolidating multiple displays into one prior to transmitting over a network, less bandwidth is required and multiple avenues for significant compression, bandwidth reduction, and bandwidth management approaches open-up, as is described below.
More specifically, this invention relates to allocating bandwidth between different sub-systems, taking into account total instantaneous available bandwidth, specific workflow, user preferences, and medical system events. Further, the invention describes systems and methods for the dynamic bandwidth management of a network, comprising a plurality of remote medical field workstations, each field workstation being configured to have the capability to take control of a given intervention as determined by workstation user roles and the dynamic needs of a specific intervention.
Further areas of applicability of the present invention will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating the preferred embodiments of the invention, are intended for purposes of illustration only, and are not intended to limit the scope of the invention. In particular, while the description focuses on applications to the local or remote control of a complex medical system, it is not so limited.
The present invention will become more fully understood from the detailed description and the accompanying drawings, wherein:
Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.
As illustrated in
Additionally, the system workstation 180 is in communication with a number of remote field workstations (FWS) 192, 194, . . . , through a network 190. At a given time, or for a given intervention, the roles of these field workstations may vary and assume that of an intervention control workstation, a teaching and training workstation, a support workstation, or a call center workstation, among other possibilities. In the context of this disclosure, the term remote is meant to indicate that the workstation is separate from the displays, UIFs, an controls typically associated with each of the sub-systems described above, such as an x-ray workstation, localization workstation, and so on. A remote workstation maybe located within the complex medical system room, in an adjacent room with view of the medical system, in a central control room within the same medical facility, or miles away at a distant facility.
It is understood that in various embodiments, the functionality of block 140 “Management of Live Remote Medical Display” described herein below may be concentrated within a specific “compression and bandwidth management processor” system, and that such process system may itself be part of the system workstation, or generally may be a component of complex medical system 100; alternatively, the functionality of block 140 may be distributed among various sub-systems, including the system workstation.
In several embodiments of the present invention, a system workstation 180 is associated with the complex medical system, and is the initial point of data and controls management and display. The system workstation configuration may be similar to the remote workstations.
In such embodiment, the system workstation 180 includes the compression and bandwidth management processor and manages the communications from the medical system 100 to remote field workstations, shown as 192 and 194. As the system workstation may itself be remote from the intervention room, it is convenient to distinguish the non-system workstation by labeling them for the purposes of the disclosure “field workstation.” It is understood that a field workstation may be physically located in a room adjacent to the system workstation room, or, in specific embodiments, may be co-located next to the system workstation in a single room. The distinction between “system workstation” (SWS) and “field workstation” (FWS) does not, however, mean that the system workstation is always the workstation actually in control of an intervention performed by the medical system 100. Indeed the systems and methods of the present invention are meant in part, to facilitate remote intervention control from one or a plurality of workstation(s) that may be several rooms away in the same building or clinic or a continent apart from the system workstation. A network 190 of limited bandwidth is available for communications between the various system and field workstations. It is clear that in practical embodiments, network communications may occur over a telecommunication wire, a coaxial wire, an RF communication, possibly including satellite links, a fiber optic cable, and other means of high-speed analog and digital telecommunications as known in the art or as may be implemented in the future. Of relevance here, is that the total bandwidth available for each system workstation-to-field workstation communication is limited, and can vary in time due to other load-sharing constraints that may exist within such a network.
This invention has been developed to enable physicians, call center representatives and other attendant staff to remotely access multiple live, high resolution displays necessary to monitor and perform medical procedures. One of the challenges addressed herein is to find a way to remotely view a number of high refresh rate, high resolution medical displays via a network with limited bandwidth requirements for broad distribution to medical centers around the world. Referring now to
In
A key part of the present invention consists of the bandwidth management for the various network connections. In that regard, the system workstation plays an essential role as interface between the interventional system sub-systems and the other workstations (FWS). The system workstation performs both information source management in the form of data compression and communication channel management through bandwidth allocation and bandwidth management. By monitoring the network, the system workstation measures the bandwidth available for each communication link to a remote field workstation. It also measures the instantaneous amounts of data sent by each of the sub-systems or information sources that it is in communication with. All of the systems and sub-systems described are also in communication to enable information or data flow back from the FWSs to the SWS, and from the SWS to the individual sub-system components that play the roles of major information sources. However in typical embodiments the volume of information flowing back to the SWS and to the individual information sources is small in comparison to that emanating from the information sources and from the SWS toward each of the FWS. Accordingly, the following the description focuses on data flows from the sub-system components to the SWS and from the SWS to the FWSs.
Different sub-systems generate information on a different scale and have different associated “native” bandwidth requirement. Although physical space as counted in number of pixels taken by a window displaying a live x-ray image could be the same as that occupied by a window showing a live ECG, the vector (1-D) dimensionality of the later versus the array (2-D) dimensionality of the former clearly points out that, in general, the bandwidth requirements associated with the live x-ray data are significantly larger. Accordingly, bandwidth allocation takes into account not only the actual display size associated to an information source, but also the intrinsic nature of the sub-system or information source.
The user of the system workstation may choose to allocate bandwidths either on an absolute or relative scale. Absolute bandwidth allocations are appropriate when experimentation has shown that for a given sub-system, below a certain bandwidth threshold meaningful transmission of information becomes compromised. Relative allocations attempt to distribute bandwidth according to workflow, priorities, sub-system intrinsic information contents, user preferences, and available instantaneous channel bandwidth.
The bandwidth allocation is also made dependent on sub-system priorities, either established as part of a protocol or examination workflow, or based on user preferences. An interventional system that includes a live, essentially real-time, endovascular imaging probe will reduce the proportional bandwidth allocation to the x-ray fluoroscopy data, all other parameters remaining the same. A given physician may attribute more weight, that is rely more, on the live fluoroscopic image than to the localization map, even as that later map is being iteratively refreshed. In one embodiment, to each composite display window associated with a specific sub-system, a slider associated uniquely to that window enables the user to manually adjust the amount of bandwidth, and therefore the image or signal quality, associated with that specific sub-system. The user described above would therefore choose a lower proportional bandwidth allocation for the localization map than for the fluoroscopy image, for instance by sliding a cursor on the slider range associated with each window. In a preferred embodiment, this relative bandwidth allocation becomes effective only when the total bandwidth available becomes lower than that required for full-fidelity transmission of all the composite information.
In a preferred embodiment, the initial window bandwidth allocation is determined automatically from design knowledge of the information sources, a combination of intervention-specific protocols, a given user history of preferences and selections, and workflow steps. The slider settings can be absolute or relative; in a preferred embodiment, the slider indicates a relative bandwidth allocation that further weights the allocation as determined by the other parameters listed above. Starting from a bandwidth-starving situation where all sub-systems operate within an allocated bandwidth insufficient for their native bandwidth requirements, as total bandwidth allocation increases for a particular network communication channel linking the SWS to a specific FWS, incremental bandwidth is allocated preferably to the sub-system whose window has the highest relative slider setting; however all linked sub-systems partake in the bandwidth increase, based on a pre-determined formula established from bandwidths parameters described above. Once the highest-priority sub-system bandwidth allocation reaches its native bandwidths, additional increases in bandwidth are allocated to the remaining sub-systems on the priority list.
Similarly, a sub-system that transmits an alert, for instance as based on one of the measured or monitored parameters, would automatically see its total share of the available bandwidth increased in proportion to its increased priority. In remote navigation of medical device, possible events of interest include the tip of a wire or catheter contacting an organ wall or tissue surface; a local ablation area reaching a certain threshold temperature; a locally measured electrical activity and activity lag as compared to a given reference falls below a threshold, and others. Multiple examples will be derived from the particular of an examination, associated diagnostic and or therapeutic goals, and interventional protocols and workflows.
An extreme example of this situation occurs when a medical event is detected and triggers an immediate set of parameter changes to the system control and associated user alarms. The medical event may be detected by simple measures, such as heart rate beat-per-minute, blood oxygen content, or more complex features, such as the detection of ventricular fibrillation. In the case of an ablation procedure, power delivery to the ablative instrument automatically stops.
Specific examination workflows are associated to specific default relative or absolute bandwidths allocations. In a preferred embodiment, a history of user preferences is built and the system workstation learns such preferences in the context of particular workflows. By design a “safe-mode” is provided should absolute bandwidth decrease between a specific remote FWS or set of remote workstations: for example, an intervention controlling workstation, at which an expert or proctoring physician is located. In such a situation, urgent communication messages are displayed, intervention steps are delayed and/or control is returned to either the SWS or a FWS with minimum bandwidth allocation sufficient for safe intervention progression. The SWS continuously monitors bandwidth allocations between FWS and in a fail-safe mode is capable of redirecting bandwidth away from secondary role FWS to the control FWS to allow for continued safe intervention.
As will be described in more details below, the specific architecture of the system considered in its entirety, comprising a first set of sub-systems that are all combined at a system workstation, and then a combined data stream from the workstation to the field workstation, enable a multi-layer approach to data compression. More specifically, at least four different compression levels or layers can be identified, and are now described in turn.
The first two compression and bandwidth management layers are described schematically in
In a key aspect of the present invention, specific bandwidth management steps and compression methods retained may be dependent upon which field workstation, or subset of field workstations, the data is intended to be sent to. As different users and different roles may accommodate or accept varying amount of compression and variable styles of bandwidth allocation, data transmission from the system workstation to the field workstations is optimized on a field workstation (or field workstation subset) basis. In such an embodiment, as represented schematically in
In another aspect of the present invention, the system workstation tracks commands or inputs from the user, and based at least partially on such inputs, modifies bandwidth allocation. As the system workstation provides a composite display, more information than can typically be viewed by a single user at any given time is presented. Accordingly at any point during an intervention, the user dynamically focuses on specific sub-systems through their associated display windows on the composite display. This and subsequent compression and bandwidth management layers are described in
Additionally in one embodiment, a means is provided at one or more workstation to track the user's eye motion. As the user is actively viewing or focusing on one area of the display, software assisted by detection means including user position sensor means and an eye-position tracking device identify the user's area of focus on the screen. The software design accounts for the rapid eye side motions that user viewing a computerized display typically experience; averaging and filtering of the eye position allows the definition of a dynamic, but relatively slowly varying area of focus.
In one embodiment, separate sub-systems windows layouts are rearranged dynamically as a function of the workflow, so that a cluster of windows provides a region of interest (typically centered on the display), wherein substantially all of the step-relevant information is found, facilitating the user focusing on the most relevant information source while maintaining secondary relevant sources in the near neighborhood of the main window. In such embodiment, according to user preferences, such dynamic layout redistribution may occur automatically, may occur only after user acceptance following a prompt (such as a voice prompt), or not occur at all. Further, several modes of layout transitions are provided that are also the object of user selection (in analogy with transition modes between slides in a PowerPoint presentation).
In any of the above described user focus center area determination means, or for any similar determination means, a focus area is determined and that determination in turn, serves as an input in the bandwidth management and compression algorithm approaches. The center focus area is allocated proportionally more bandwidth as compared to the surrounding areas. Transition between these center and surrounding areas may be effected in a linear or non-linear manner; and maybe relatively marked or, in the limit, non-existent.
In one embodiment, spatially encoding data with a center focus comprises sampling the data on a radial grid centered at the focus point. Accordingly, the density of samples is the highest at the focus center and decreases linearly with distance from the center.
Thus, and referring to
In a fourth layer of compression denoted by 420 in
In a preferred embodiment, further bandwidth gains are achieved by using both spatial and temporal compression method, whereby bandwidth is allocated both based upon spatial and temporal resolution considerations. In one embodiment, both the spatial and temporal resolution of the data away from the user focus center are reduced as compared to the focus center. Alternatively, define different spatial horizons with different spatial/time resolution trade-offs. That is, the time sampling frequency is spatially dependent; or stated conversely, the spatial resolution is also a function of the time resolution. This compression and bandwidth management layer is represented schematically by block 430 in
In a typical intervention involving several users with a subset of users at a set of remote workstations, roles are typically assigned for all or most of the intervention to the different users. Control of the intervention is within the hands of the intervening physician; occasionally a mentoring physician or expert may for a few moments or for more extended period of times, be in control of the interventional system. In any case, a limited subset of users is actually in control of the intervention, and other users are present in a more limited role. For this subclass of user, it might be in general, acceptable to provide either a somewhat reduced spatial or temporal resolution. Alternatively it is acceptable to provide a restored image quality at the cost of a small lag in the presentation of images and data. In such a situation, and for the subclass of users who don't need real-time data, send high spatial resolution at a low frame rate, and up-interpolate temporally between the frames: introduces a bit of a lag, of the order of 1 or a few seconds. There composite images are sent at a high-spatial resolution, but a reduced frame rate. Images are up-interpolated at the receiving workstation 440, to provide the user with a smooth flowing viewing experience. Accordingly, this defines a fifth level of compression denoted by 430 in
Thus in a general sense, the three parameters available for data compression and bandwidth management: 1) spatial resolution; 2) temporal resolution; 3) amount of lag acceptable. As seen above, these parameters are in general considered in combination, and in the context of this disclosure referred to individually or in various combinations as information coding methods.
Compression levels two to five, and in particular, compression levels two, three, and five, depend upon specific parameters set by the user(s) of the receiving workstation 440. Accordingly, communications means 450 is provided for parameter and feedback information communication from the remote field workstation to each of the compression levels, as shown respectively by arrows 452, 454, 456, 458, and 460. Focusing in particular on block 460, showing the weights calculations for compression and bandwidth management level two, and as described above, the compression weights 462, . . . , 464 depend on the workstation role, user preferences, user focus center, bandwidth available, and so forth as described previously. Further, as indicated in block 460, these calculations are dynamic as most if not all of these parameters may vary over the course of one medical system intervention.
In a typical interventional system embodiment, most sub-systems generate data at a predictable rate. As an example, the data rate for an x-ray fluoroscopy sequence is given by the image size and pixel digitization depth, data acquisition rates, and apart from being turned on or off, is unlikely to change over time. This provides a convenient means to monitor the information content of each source, by comparing the nominal information rate to the actual information rate achieved by compressing each source data stream. Accordingly, a means is provided for each sub-system to monitor its relative information rate and absolute bandwidth needs as a function of time. The compression and bandwidth management processor thus knows of these absolute and relative bandwidth requirements and can reallocate bandwidth over time as the information source requirements change. Alternatively or additionally to the comparative means proposed above, other means of calculating the absolute and relative bandwidth requirements of an information source may be used, as is know in the art. The absolute and relative bandwidth requirements of an information source are then taken into account by the system workstation in generating a field-workstation-dependent bandwidth management.
As illustrated schematically in
It is common for physicians during an intervention to review data to ascertain the progress of the procedure, and ensure that the desired results have been achieved. One form of review consists of looking at image data sequence acquired after a specific intervention step. In one embodiment of the present invention, error data sequence are collected and measured on the encoding (system) workstation; the error being defined as the difference between the uncompressed data being presented and displayed at the system workstation and the data being transmitted to a specific field workstation; it is therefore FWS-specific. Although total network allocated bandwidth might vary slowly, it is reasonable to expect that the composite display, or at least sub-components thereof, will see fast temporal variations. Accordingly, when the bandwidth requirements fall, or conversely when the amount of available bandwidth increases, the past error sequences are automatically sent to one or a set of field workstations, and a sequence of exact image sequences is reconstructed in the background at those FWSs. Should the user/physician reach a point where image review is necessary, it becomes possible to then present at least part of a past sequence of images at full, native, image quality. This facilitates the physician review, and provides the best available image quality for monitoring the effect of the intervention up to now and controlling the intervention next steps.
An exemplary workflow of the systems and methods of one embodiment of the present invention is described schematically in
The advantages of the above described embodiments and improvements should be readily apparent to one skilled in the art, as to enabling bandwidth management and compression levels determination for the transmission of complex, composite sub-system data over a network of limited bandwidth to a set of remote users at a number of field workstations. Additional design considerations may be incorporated without departing from the spirit and scope of the invention. Accordingly, it is not intended that the invention be limited by the particular embodiments or forms described above, but by the appended claims.
This application claims priority to U.S. Provisional Patent Application Ser. No. 60/948,311, filed Jul. 6, 2007. The disclosure of the above-referenced application is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60948311 | Jul 2007 | US |
