Patent Application
20170319282
Embodiments described herein generally relate to network communications and in particular, to systems and methods for integrated user environments.
Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and deleterious side effects. Teleoperated surgical systems that use robotic technology (so-called surgical robotic systems) can be used to overcome limitations of manual laparoscopic and open surgery. Advances in telepresence systems provide surgeons views inside a patient's body, an increased number of degrees of motion of surgical instruments, and the ability for surgical collaboration over long distances. In view of the complexity of working with teleoperated surgical systems, proper and effective training is important.
In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which:
The following description is presented to enable any person skilled in the art to create and use systems and methods of a medical device simulator. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the inventive subject matter. Moreover, in the following description, numerous details are set forth for the purpose of explanation. However, one of ordinary skill in the art will realize that the inventive subject matter might be practiced without the use of these specific details. In other instances, well-known machine components, processes and data structures are shown in block diagram form in order not to obscure the disclosure with unnecessary detail. Flow diagrams in drawings referenced below are used to represent processes. A computer system can be configured to perform some of these processes. Modules or subsystems within flow diagrams representing computer implemented processes represent the configuration of a computer system according to computer program code to perform the acts described with reference to these modules. Thus, the inventive subject matter is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
Surgical training can come in various forms, including observation, practice with cadavers or surgical training models, and simulation training. In the field of teleoperated surgery, all of these training techniques can be used. In order to provide a consistent and repeatable experience, simulation training provides distinct advantages.
When analyzing performance for a teleoperated simulator, instructional objectives can be viewed on a continuum with basic system skills on one end of the continuum and robotic surgical procedures on the other end. In the middle, robotic surgical skills and tasks are represented. Thus a user can begin learning with basic robotic system skills, such as dexterous tasks like needle targeting, moving objects, or navigating instruments in space. Eventually, the user can progress to the middle of the continuum and practice robotic surgical skills, such as suturing or knot tying. After gaining proficiency in skills, the user can progress to robotic surgical procedures and procedural tasks, such as a hysterectomy.
Simulation training can be provided to a user in various modes. The user can participate in individual training modules attempting a training task with or without guidance. Such guidance can be provided by the training module, for example, with audio prompts, textual overlays, or the like. Alternatively, the user can participate in a cooperative environment with an expert user (e.g., proctor, instructor, or teacher) providing guidance. Systems and processes illustrated herein describe a cooperative environment where one or more remote users can view an expert user's movements and annotations. Such an expert-guided experience can improve education and reduce training time.
In alternative embodiments, the teleoperated surgical system 100 can include more than one manipulator assembly 102. The exact number of manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room among other factors.
The master assembly 110 can be located in the same room as the operating table 108. However, it should be understood that the surgeon 112 can be located in a different room or a completely different building from the patient 106. The master assembly 110 generally includes one or more control device(s) 114 for controlling the manipulator assembly 102. The control device(s) 114 can include any number of a variety of input devices, such as gravity-balanced arms, joysticks, trackballs, gloves, trigger grips, hand-operated controllers, hand motion sensors, voice recognition devices, eye motion sensors, or the like. In some embodiments, the control device(s) 114 can be provided with the same degrees of freedom as the associated surgical instruments 104 to provide the surgeon 112 with telepresence, or the perception that the control device(s) 114 are integral with the instrument 104 so that the surgeon 112 has a strong sense of directly controlling the instrument 104. In some embodiments, the control device 114 is a manual input device that moves with six degrees of freedom or more, and which can also include an actuatable handle or other control feature (e.g., one or more buttons, switches, etc.) for actuating instruments (for example, for closing grasping jaws, applying an electrical potential to an electrode, delivering a medicinal treatment, or the like).
A visualization system 116 provides a concurrent two- or three-dimensional video image of a surgical site to surgeon 112. The visualization system 116 can include a viewing scope assembly. In some embodiments, visual images can be captured by an endoscope positioned within the surgical site. The visualization system 116 can be implemented as hardware, firmware, software, or a combination thereof, and it interacts with or is otherwise executed by one or more computer processors, which can include the one or more processors of a control system 118.
A display system 120 can display a visual image of the surgical site and surgical instruments 104 captured by the visualization system 116. The display system 120 and the master control devices 114 can be oriented such that the relative positions of the visual imaging device in the scope assembly and the surgical instruments 104 are similar to the relative positions of the surgeon's eyes and hands so the operator (e.g., surgeon 112) can manipulate the surgical instrument 104 with the master control devices 114 as if viewing a working volume adjacent to the instrument 104 in substantially true presence. By “true presence” it is meant that the presentation of an image is a true perspective image simulating the viewpoint of an operator that is physically manipulating the surgical instruments 104.
The control system 118 includes at least one processor (not shown) and typically a plurality of processors for effecting control between the surgical manipulator assembly 102, the master assembly 114, and the display system 116. The control system 118 also includes software programming instructions to implement some or all of the methods described herein. While control system 118 is shown as a single block in the simplified schematic of
In some embodiments, the control system 118 can include servo controllers to provide force and torque feedback from the surgical instrument 104 to the master assembly 114. Any suitable conventional or specialized servo controller can be used. A servo controller can be separate from, or integral with, the manipulator assembly 102. In some embodiments, the servo controller and the manipulator assembly 102 are provided as part of a robotic arm cart positioned adjacent to the patient 106. The servo controllers transmit signals instructing the manipulator assembly 102 to move the instrument 104, which extends into an internal surgical site within the patient body via openings in the body.
Each manipulator assembly 102 supports at least one surgical instrument 104 (e.g., “slave”) and can comprise a series of non-teleoperated, manually articulatable linkages and a teleoperated robotic manipulator. The linkages can be referred to as a set-up structure, which includes one or more links coupled with joints that allows the set-up structure to be positioned and held at a position and orientation in space. The manipulator assembly 102 can be driven by a series of actuators (e.g., motors). These motors actively move the robotic manipulators in response to commands from the control system 118. The motors are further coupled to the surgical instrument 104 so as to advance the surgical instrument 104 into a naturally or surgically created anatomical orifice and move the surgical instrument 104 in multiple degrees of freedom that can include three degrees of linear motion (e.g., X, Y, Z linear motion) and three degrees of rotational motion (e.g., roll, pitch, yaw). Additionally, the motors can be used to actuate an effector of the surgical instrument 104 such as an articulatable effector for grasping tissues in the jaws of a biopsy device or an effector for obtaining a tissue sample or for dispensing medicine, or another effector for providing other treatment as described more fully below. For example, the instrument 104 can be pitched and yawed around the remote center of motion, and it can be inserted and withdrawn through the remote center of motion (e.g., the z-axis motion). Other degrees of freedom can be provided by moving only part of the instrument (e.g., the end effector). For example, the end effector can be rolled by rolling the shaft, and the end effector is pitched and yawed at a distal-end wrist.
In an embodiment, there are two master controllers 204, each with two finger loops 210 for which the user may insert an index finger and thumb of a respective hand. The two master controllers 204 may each control a virtual surgical instrument. The user may be provided software or hardware mechanisms to swap between multiple instruments for one or both master controller 204. For example, a user may be provided three instruments, such as two forceps and a retractor. One or both of the forceps may be an energy instrument able to cauterize tissue. The user may first use the forceps at each master controller 204, then switch the right master controller 204 to control the retractor to expose a section of the surgical field, and then switch the right master controller 204 back to the forceps to continue cutting, probing, or dissecting tissue.
While using the master controllers 204, the user is provided with full three-dimensional range of motion (x, y, and z axis) along with rotational motion (roll, pitch, yaw) in addition to pinching motion with the index and thumb (or any two fingers inserted into the loops 210). As such, by moving the appropriate master controller 204, the user is able to manipulate the corresponding surgical instrument through a full range of motion.
In an embodiment, the display system 120 can display a virtual environment simulating a surgical site within a patient. The virtual environment can include various biological structures in addition to the surgical instrument 104. The surgeon 112 operates the instrument 104 within the virtual environment to train, obtain certification, or experiment with various skills or procedures without having the possibility of harming a real patient. Simulating surgical procedures also has the advantage of requiring fewer components. For example, a patient-side cart is not needed because there is no actual patient. Thus, simulation provides increased convenience and accessibility.
Disclosed herein is a virtual training environment that includes a local user's virtual surgical instruments rendered in a virtual surgical environment along with an expert user's surgical instruments. One goal is to obtain more consistent training outcomes. Another goal is to reduce training time. Yet other goals include, but are not limited to, providing a more engaging and interactive training environment and providing a platform for expert feedback to increase training efficacy.
The expert user's surgical instruments can be presented in a translucent or semi-transparent overlay in the trainee's screen (represented by the dashed outline virtual instruments in the combined scene 404). In this manner, the expert user who is operating a separate master assembly is able to visually guide or advise the trainee user and the trainee can mimic or watch the expert's virtual instruments in the display system. Other visual effects can be applied to the expert user's surgical instruments, such as a semi-transparent effect, see-through effect, or an abstracted representation (e.g., a dotted outline, ghosted shape, cartoon drawing, etc.). Optionally, in some embodiments, the expert user's surgical instruments are rendered in a manner to resemble the trainee user's virtual surgical instruments (e.g., opaque, shaded, etc.). Further, while some embodiments are described with the expert's virtual surgical instruments being visually modified (e.g., using a semi-transparent effect), it is understood that such modifications can be applied to the trainee user's virtual instruments. For example, in an embodiment, at the expert user's station, the expert user's virtual instruments are rendered as opaque while the trainee's virtual instruments are rendered as semi-transparent or see-through. Additionally, the effect used on the virtual instrument (either trainee or expert) can be modified before or during an exercise. The modifications can be used to improve training methodologies.
In a similar fashion, some or all of the input data received at the expert system 502 as a result of a user's operation of the expert system 502 is transmitted to the trainee system 500. At the expert system 502, the input data received at the expert system 502 as a result of the user's operation of the expert system 502 is used to render a local scene 512 (local with respect to the user at the expert system 502). The input data received at the expert system 502 as a results of the user's operation of the expert system 502 is transmitted (arrow 514) to the trainee system 500 and rendered as a remote scene 516 (remote with respect to the trainee system 500).
The trainee system 500 renders a composite scene 518 that includes the local scene 508 and the remote scene 516. The composite scene 518 may alter the remote scene 516 using various graphical manipulations, for example making the remote scene 516 translucent, changing the color of the remote virtual instruments, or other enhancements to allow the user of the trainee system 500 to more easily distinguish the local virtual surgical instruments from the remote (e.g., expert) surgical instruments in the composite scene 518. The expert system 502 produces a similar composite scene 520 to provide the expert system 502 user a view of the local and remote virtual surgical instruments. The expert system 502 can optionally alter the local scene 512 or the remote scene 510 (local and remote from the perspective of the expert system 502) using various graphical manipulations, for example by making the local scene 512 or remote scene 510 translucent, semi-transparent, changing the color of the virtual instruments, etc.
It is understood that more than one remote master assembly 110 can receive the input control signals from the communication subsystem 606 and that the communication subsystem 606 can receive input control signals from more than one remote master assembly 110. In this manner, several instructors may provide concurrent instruction or guidance to a local user, each instructor having virtual surgical instruments represented in the local user's display. Also in this manner, several trainee users may receive instruction from one or more instructors. While
The communication subsystem 606 can communicate with the remote master assembly 110 using various networking protocols or technologies, such as a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-A or WiMAX networks).
The environmental variable can be represented as a data structure of one or more n-tuples. For example, the n-tuple can be a 4-tuple as (input_id, x-position, y-position, z-position). In some embodiments, the input_id is used to uniquely identify an input of a user interface of a teleoperated surgical system. For example, the value “1” can correspond to a left master controller and the value “2” can correspond to a right master controller. As such, the 4-tuple of (1, 33.4, 24.9, 18.4) represents that the position of the left master controller is 33.4 cm in the x-position, 24.9 cm in the y-position, and 18.4 cm in the z-position. The master assembly can translate the x-y-z position into a corresponding position in the virtual environment to correctly represent the position, attitude, or speed of a virtual surgical instrument corresponding to the left master controller. The same 4-tuple can be used locally to render a local scene or transmitted to a remote master controller of a teleoperated surgical system to render a scene. Transmitting the n-tuple is advantageous in that it reduces network load and decreases latency.
In another embodiment, the pose of the master controller in addition to its x-y-z position is transmitted from one master assembly to another. This gives the orientation of the wrist. The value of open/close of the instrument pincher formation is also transmitted. A 4×4 transform matrix with a 3×3 rotation matrix in the upper left and a 3×1 translation vector in the upper right is used. In addition, the input_id indicates left/right hand, which remote user it is (in the case where there are multiple remote users), and the open/close position of the grippers (between 0 and 1, with 0 being fully open and 1 being fully closed) are transmitted.
In an embodiment, the second user interface includes a master controller operated by a user of the second user interface, and the environmental variable includes a position, speed, or rotation of the master controller.
In an embodiment, the communication subsystem 606 is further configured to receive an annotation variable from the second user interface, the annotation variable describing an annotation to render on the composite scene. In such an embodiment, the video subsystem 602 is further configured to composite the composite scene to include the annotation and present the annotation in the composite scene. In an embodiment, the annotation includes a crayon control, a high lighter control, or a pointer icon. For example, the remote user (e.g., proctor, instructor, teacher, etc.) can use a master controller 204 to control a crayon icon to draw arrows, circles, dashes, etc. on the shared screens in order to annotate them. Annotations can be provided as text, figures (e.g., circles, squares, etc.), free-form drawing, pictures, icons, or the like. The annotation can be selected by a user of the second user interface.
Annotations can be rendered in the world coordinate frame so that they are tied to the environment and not to a particular camera reference frame. In this configuration, annotations are able to persist at a given location in the environment regardless of changes in camera angle. For example, an expert can annotate a dot on a suture sponge that the trainee is to focus on during practice, where the dot maintains a persistent location on the sponge during the exercise regardless of camera changes.
In an embodiment, the environmental variable includes a camera control variable. In such an embodiment, the video subsystem 602 is further configured to render the local scene includes rendering the local scene using the camera control variable.
In an embodiment, the local scene includes a virtual surgical instrument controlled by the user of the first user interface.
In an embodiment, the video subsystem is further configured to render the remote scene as a translucent layer, the translucent layer allowing the user of the first user interface to view the local scene when viewing the composite scene.
In an embodiment, the master assembly can include a training subsystem to provide a surgical exercise to the user of the first user interface, where the surgical exercise is also substantially concurrently provided to a user of the second user interface.
In an embodiment, the communication subsystem 606 is configured to receive the environmental variable over a wide-area network. In an embodiment, the wide-area network comprises the Internet. In an embodiment, the wide-area network comprises a wireless network.
In an embodiment, the video subsystem 602 is configured to render the local scene and the remote scene on separate canvases.
In an embodiment, the video subsystem 602 is configured to render the composite scene on a separate canvas from the rendering the local scene.
In an embodiment, receiving the environmental variable comprises receiving the environmental variable over a wide-area network. In an embodiment, the wide-area network comprises the Internet. In an embodiment, the wide-area network comprises a wireless network.
At block 704, a local scene is rendered at the first user interface, the local scene representing a state of operation of the first user interface.
At block 706, a remote scene is rendered at the first user interface, the remote scene representing a state of operation of the second user interface and the remote scene based at least in part on the environmental variable. In an embodiment, rendering the remote scene comprises rendering the remote scene as a translucent layer, the translucent layer allowing the user of the first user interface to view the local scene when viewing the composite scene.
At block 708, the local scene and the remote scene is composited to produce a composite scene. In an embodiment, the local scene includes a virtual surgical instrument controlled by the user of the first user interface. In an embodiment, rendering the local scene and rendering the remote scene are performed on separate canvases.
At block 710, the composite scene is presented to a user of the first user interface. In an embodiment, rendering the composite scene is performed on a separate canvas from the rendering the local scene.
In an embodiment, the method 700 includes receiving an annotation variable from the second user interface, the annotation variable describing an annotation to render on the composite scene; and compositing the composite scene to include the annotation; where presenting the composite scene includes presenting the annotation in the composite scene. In an embodiment, the annotation includes a crayon control, a high lighter control, or a pointer icon. In an embodiment, the annotation is selected by a user of the second user interface.
In a further embodiment, the method 700 includes providing a surgical exercise to the user of the first user interface, and wherein the surgical exercise is also substantially concurrently provided to a user of the second user interface.
Example computer system 800 includes at least one processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both, processor cores, compute nodes, etc.), a main memory 804 and a static memory 806, which communicate with each other via a link 808 (e.g., bus). The computer system 800 may further include a video display unit 810, an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In one embodiment, the video display unit 810, input device 812 and UI navigation device 814 are incorporated into a touch screen display. The computer system 800 may additionally include a storage device 816 (e.g., a drive unit), a signal generation device 818 (e.g., a speaker), a network interface device 820, and one or more sensors (not shown), such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor.
The storage device 816 includes a machine-readable medium 822 on which is stored one or more sets of data structures and instructions 824 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 824 may also reside, completely or at least partially, within the main memory 804, static memory 806, and/or within the processor 802 during execution thereof by the computer system 800, with the main memory 804, static memory 806, and the processor 802 also constituting machine-readable media.
While the machine-readable medium 822 is illustrated in an example embodiment to be a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 824. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, by way of example, semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
The instructions 824 may further be transmitted or received over a communications network 826 using a transmission medium via the network interface device 820 utilizing any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-A or WiMAX networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.
It will be appreciated that, for clarity purposes, the above description describes some embodiments with reference to different functional units or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors or domains can be used without detracting from the present disclosure. For example, functionality illustrated to be performed by separate processors or controllers can be performed by the same processor or controller. Hence, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.
Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. One skilled in the art would recognize that various features of the described embodiments can be combined in accordance with the present disclosure. Moreover, it will be appreciated that various modifications and alterations can be made by those skilled in the art without departing from the spirit and scope of the present disclosure.
In addition, in the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.
The foregoing description and drawings of embodiments in accordance with the present invention are merely illustrative of the principles of the inventive subject matter. Therefore, it will be understood that various modifications can be made to the embodiments by those skilled in the art without departing from the spirit and scope of the inventive subject matter, which is defined in the appended claims.
Thus, while certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad inventive subject matter, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications can occur to those ordinarily skilled in the art.
This patent application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application 62/079,392, entitled “INTEGRATED USER ENVIRONMENTS,” filed Nov. 13, 2014, which is incorporated by reference herein in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US15/60298 | 11/12/2015 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62079392 | Nov 2014 | US |
