Patent Grant
11039734
The invention relates to a depiction system for generating a real time depiction of movements of a surgical tool during a minimally invasive surgery and adapted for in real time providing a surgeon with information, such as information about the position of the surgical tool.
Minimally invasive surgery has been used increasingly in recent years due to the benefits compared to conventional open surgery as it reduces the trauma to the patient tissue, leaves smaller scars, minimizes post-surgical pain and enables a faster recovery of the patient.
For example, in laparoscopic surgery which is a typical form of minimally invasive surgery the surgeon accesses a body cavity, such as the abdominal or pelvic cavity, through a series of small incisions. A laparoscope is inserted through an incision, and conventionally connected to a monitor, thereby enabling the surgeon to see the inside of the abdominal or pelvic cavity. In order to perform the surgical procedure, surgical instruments are inserted through other incisions. In addition, the body cavity around the surgical site is inflated with a fluid, preferably gas e.g. carbon dioxide in order to create an ‘air’ space within the cavity to make space for the surgeon to view the surgical site and move the laparoscopic instruments.
Invasive surgery procedures are generally performed through openings in a patient's skin—often relatively small openings—and the surgical site is visualized for the surgeon by inserting a camera, such as an endoscope into the body cavity and displaying the images on a screen.
In order to improve the vision for surgeon, in particular to make it easier for the surgeon to determine the sizes of various organs, tissues, and other structures in a surgical site, several in-situ surgical metrology methods have been provided in the prior art. Different types of optical systems have been applied to provide an improved vision of the surgical site, which is approaching a 3D vision.
US 2013/0296712 describes an apparatus for determining endoscopic dimensional measurements, including a light source for projecting light patterns on a surgical sight including shapes with actual dimensional measurements and fiducials, and means for analyzing the projecting light patterns on the surgical site by comparing the actual dimensional measurements of the projected light patterns to the surgical site.
WO 2013/163391 describes at system for generating an image, which the surgeon can use for measuring the size of or distance between structures in the surgical field by using an invisible light for marking a pattern to the surgical field. The system comprises a first camera; a second camera; a light source producing light at a frequency invisible to human eye; a dispersion unit projecting a predetermined pattern of light from the invisible light source; an instrument projecting the predetermined pattern of invisible light onto a target area; a band pass filter directing visible light to the first camera and the predetermined pattern of invisible light to the second camera; wherein the second camera images the target area and predetermined pattern of invisible light, and computes a three-dimensional image.
US2008071140 discloses an endoscopic surgical navigation system comprises a tracking subsystem to capture data representing positions and orientations of a flexible endoscope during an endoscopic procedure, to allow co-registration of live endoscopic video with intra-operative and/or pre-operative scan images. Positions and orientations of the endoscope are detected using one or more sensors and/or other signal-producing elements disposed on the endoscope.
US2010268067 disclose methods, systems, devices, and computer-readable media for image guided surgery for allowing a physician to use multiple instruments for a surgery and simultaneously provide image-guidance data for those instruments.
US2011069159 discloses a system for orientation assistance and display of an instrument that is inserted or present in the natural or artificially produced hollow cavity (human, animal, object), and that is equipped with one or more sensor units. Multiple measurements of the 3D position of the instrument equipped with one or more sensor units are performed by positioning a measuring system, so that a precise orientation and positioning of the instrument in the body can be computed. The 3D position data are used to compute a virtual image of the instrument synchronously. The virtual images are then either projected directly in exact position onto the body surface of a person or combined in a body surface image (real video camera image of the patient) onto a monitor or superimposed (virtual or augmented reality).
It has also been suggested to generate augmented reality vision of surgery cavities for providing an improved view of internal structures of patient organs of to determine minimally distance to a cavity surface or organ of a patient. Such systems are described in the articles “Augmented reality in laparoscopic surgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20 (2011) 189-201 and “An effective visualization technique for depth perception in augmented reality-based surgical navigation” by Choi Hyunseok et al. The international journal of medical robotics and computer assisted surgery, 2015 May 5. doi: 10.1002/rcs.1657.
US2014163359 discloses a surgical tracking system for assisting an operator to perform a laparoscopic surgery of a human body. The surgical tracking system comprising: a) at least one endoscope adapted to acquire real-time images of a surgical environment within said human body; b) a maneuvering subsystem adapted to control the spatial position of said endoscope during said laparoscopic surgery; and, c) a tracking subsystem in communication with said maneuvering subsystem, adapted to control the maneuvering system so as to direct and modify the spatial position of said endoscope to a region of interest. The system generates real life correlated images of movement of a surgical tool.
In an embodiment of the depiction system of the invention the system provides an alternative system for generating good visibility of at least a part of a body cavity during minimally invasive surgery in particular with respect to providing good visual information to the surgeon about the position of a surgical instrument relative to the surgical site.
The depiction system of the invention has shown to be capable of providing a surprisingly good visibility to a user, such as a surgeon or a person training for performing a surgery. In an embodiment the system of the present invention aims to provide a depiction comprising a projection of the position of a surgical tool rather than imaging the surgical tool. Thereby the user can be provided with a large amount of information in one single depiction. Whereas one single depiction can provide the use with large amount of information, the invention also comprises a system generating two or more depictions or images as further described below.
The depiction system of the invention is suitable for generating a real time correlated depiction of movements of a surgical tool.
The system comprising
At least one of the 3D surface data generation means and the position data generation means comprises surface position data generation means for providing surface position data of at least the target area.
The computer system is programmed for
transmitting the depiction data to the display unit for real time correlated depiction of movements of the surgical tool.
The computer system may comprise one single computer or a plurality of computers in data connection, wireless, by wire and/or via the internet.
The minimally invasive surgery cavity may be a patient or an artificial cavity. In general, an artificial cavity is useful in a training procedure. In other procedures as well as in supervised procedures of relatively unexperienced surgeons the minimally invasive surgery cavity is advantageously a minimally invasive surgery cavity of a patient such as a human or an animal.
The minimally invasive surgery cavity may in principle be any kind of body cavities including naturally occurring cavities as well as cavities formed by expansion of a non-luminal part of a body (pseudo cavity). Examples of suitable minimally invasive surgery cavities includes an abdominal cavity (the area between the diaphragm and the brim of the pelvis), a chest cavity, abdominopelvic cavity (contains both the abdominal and pelvic cavities), cranial cavity (contains the brain), diaphragm (muscle that separates the thoracic and abdominopelvic cavities), mediastinum (central area within the thoracic cavity), pelvic cavity (contains the reproductive organs and urinary bladder), pericardial cavity (contains the heart), pleural cavity (contains the lungs), thoracic cavity (enclosed by the ribs, sternum, and vertebral column), vertebral cavity (contains the spinal cord), etc.
Usually the minimally invasive surgery procedure includes that the surgeon provides access to the surgery site by an incision and apply a cannula (sometimes also called a sleeve) to provide an access port through the incision. The cannula functions as a portal for the subsequent placement of a surgical instrument comprising a surgical tool. The term ‘surgical tool’ is herein used to designate the distal part of a surgical instrument adapted to be inserted into the minimally invasive surgery cavity. A surgical instrument usually has a distal end and a proximal end and comprises a handle portion at its proximal end, a surgical tool at its distal end and a body portion connecting the handle portion to the surgical tool. The surgical tool may for example be graspers, scissors, staplers, etc. In an embodiment the body portion of the surgical instrument is considered as a part of the surgical tool. For examples sensors mounted to the surgical tool portion may be mounted to the body portion of the surgical instrument.
The cannula usually comprises one or more seals to seal against gas slip-out and to accommodate an instrument. After the cannula has been inserted, the minimally invasive surgery cavity is usually enlarged by blowing a gas into the cavity thereby providing a cavity sufficiently large for performing the minimally invasive surgery procedure.
Often the surface of the minimally invasive surgery cavity is much curved. The term ‘target area’ of the surface of the minimally invasive surgery cavity is herein used to designate the area which the surgeon has focus on, and the target area may advantageously comprise a surgery site and/or a surface area which potentially could be in risk of damage during the surgery procedure for example a critical structure, such a vene structure. The depiction data representing said depiction of said real tile spatial positions of the surgical tool onto at least a portion of the surface contour of the surface section of the minimally invasive surgery, where the portion of the surface contour need not comprise the surface contour of the target area constantly during a minimally invasive surgery procedure. It is sufficient that the contour of the target area forms part of the portion of the surface contour when the surgical tool is approaching the target area.
In an embodiment depiction system is configured for determining real time spatial position data comprising determining a distance from the surgical tool to a critical structure of the surface section.
The phrase “real time” is herein used to mean the time it requires the computer to receive and process constantly changing data optionally in combination with other data, such as predetermined data, reference data, estimated data which may be non-real time data such as constant data or data changing with a frequency of above 1 minute to thereby provide a depiction of corresponding actual changes as they occur or within up to 5 seconds, preferably within 1 second, more preferably within 0.1 second of occurrence.
The terms distal and proximal should be interpreted in relation to the orientation of the surgical tool i.e. the distal end of the surgical tool is the part of the surgical tool farthest from the incision through which the surgical instrument comprising the surgical tool is inserted.
The phrase “distal to” means “arranged at a position in distal direction to the surgical tool, where the direction is determined as a straight line between a proximal end of the surgical tool to the distal end of the surgical tool. The phrase “distally arranged” means arranged distal to the distal end of the surgical tool.
The term “substantially” should herein be taken to mean that ordinary product variances and tolerances are comprised.
The term “about” is generally used to ensure that what is within measurement uncertainties are include. The term “about” when used in ranges, should herein be taken to mean that what is within measurement uncertainties are included in the range.
It should be emphasized that the term “comprises/comprising” when used herein is to be interpreted as an open term, i.e. it should be taken to specify the presence of specifically stated feature(s), such as element(s), unit(s), integer(s), step(s) component(s) and combination(s) thereof, but does not preclude the presence or addition of one or more other stated features.
Throughout the description or claims, the singular encompasses the plural unless otherwise specified or required by the context.
The portion, i.e. the area extension of the surface contour which is comprised in the depiction data calculation may in principle have any size larger than zero. In order to provide a depiction containing desirably large amount of information for the user, the portion of the surface contour advantageously has a size which is sufficiently large to reflect the surface contour of the portion. In an embodiment the portion of the surface contour has a size of at least about 5 mm2, such as at least about 10 mm2, such as at least about 1 cm2, such as at least about 5 cm2, such as at least about 10 cm2, such as at least about 25 cm2 determined as the maximal size of a projection of the surface contour onto a plan—i.e. the plan is selected as the plan providing the largest projected surface. In an embodiment the portion of the surface contour has a size as least as large as the target area and/or as least as large as the surgical site.
Advantageously the portion of the surface contour which is comprised in the depiction data calculation has a size which is dynamically changing in dependence of the real time position data, Thereby the depiction may reveal any movements of the surgical tool.
In an embodiment the portion of the surface contour is increasing with increasing distance between the spatial position the part of the surgical tool and the surface section.
In an embodiment the 3D surface data generation means and/or the position data generation means is/are part of the depiction system and advantageously the 3D surface data generation means and/or the position data generation means is/are controlled by the computer system. The computer system is preferably programmed to control the operation of the 3D surface data generation means and/or the position data generation means.
The 3D surface data generation means may be any means capable of providing the computer system with the 3D surface data, such as data from a database, data supplied to the computer system by a user, e.g. in form of pre-measured data e.g. by scanning e.g. CT scanning e.g. MR or real-time measured data.
In an embodiment the 3D surface data generation means comprises a 3D database system. The 3D database system comprises at least one 3D data set for at least one surface section of each of a plurality of classified minimally invasive surgery cavities, wherein each 3D data set is associated to the respective surface section(s) of at least one of the classified minimally invasive surgery cavities. Preferably the computer is configured for acquiring a 3D data set for at least one surface section of a classified minimally invasive surgery cavity.
The classified minimally invasive surgery cavities are preferably represented by their respective classification. The classification may in principle comprise any classification data that describes the classified minimally invasive surgery cavities e.g. type of cavity, age of patient, weight of patient, gender of patient, height, body circumferential dimension(s) or any combinations comprising the abovementioned or any other specific or average or level of patient data.
In an embodiment the computer is programmed to receive instructions for selecting a 3D data set and based on an instruction to acquire the selected 3D data set from the 3D database. The instruction may advantageously comprise classification data. The instruction may for example be given to computer system via a digital user interface and/or orally by a user.
Unless otherwise stated, the term “user” is herein used to include any user, such as a surgeon, an assisting surgeon, a supervisor, a computer programmed to act as supervisor, an artificially intelligent computer and/or a robot, where a robot is programmed to perform at least one surgeon duty, e.g. to perform or assist in a minimally invasive surgery procedure.
In an embodiment the surgeon is a robotic surgeon.
In an embodiment the visual information means optical data as received by a robotic surgeon and representing a surface section of the minimally invasive surgery cavity.
In an embodiment the computer is programmed to receive instructions comprising classification data for selecting several 3D data sets and based on this instruction to acquiring selected 3D data sets and process the acquired 3D data sets to determine the resulting 3D data which is applied to determine the 3D surface contour of the surface section of the minimally invasive surgery cavity.
In an embodiment the computer is programmed to receive instructions comprising classification data for selecting several 3D data sets and based on this instruction to acquiring selected 3D data sets and processing the acquired 3D data sets to determine the resulting 3D data which is applied to determine the 3D surface contour of the surface section of the minimally invasive surgery cavity.
The database system is data connectable or in data connection with the computer e.g. by wireless connection, by wire connection, via the internet and/or by being loaded onto a memory of the computer system.
The database system may continuously be build up or updated by adding new 3D data sets to the database.
In an embodiment the database system is loaded into a memory in data communication with or being a part of the computer system.
The 3D data sets of the database may be obtained by any methods and will usually be 3D data sets from previously performed minimally invasive surgery procedures and/or estimated and/or calculated 3D data sets and/or 3D data sets obtained by a scanning such as described further below.
Advantageously one or more of the 3D data sets comprises estimated 3D data, calculated 3D data, measured 3D data or any combination thereof. Preferably the 3D data sets each are associated to a patient characteristic, such as a patient characteristic comprising age, gender, weight, height body circumferential dimension(s) or any combinations comprising the abovementioned.
The estimated 3D data, calculated 3D data and/or measured 3D data of the 3D data sets may be obtained or generate by any methods, such as methods known from prior art and/or methods described herein for obtaining 3D data a surface section of a minimally invasive surgery cavity. In particularly ultra sound based methods, CT scanning and/or MRI are suitable as well as the prior art methods as disclosed in the articles “Augmented reality in laparoscopic surgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20 (2011) 189-201 and “An effective visualization technique for depth perception in augmented reality-based surgical navigation” by Choi Hyunseok et al. The international journal of medical robotics and computer assisted surgery, 2015 May 5. doi: 10.1002/rcs.1657.
In an embodiment the 3D surface data generation means comprises a user input data connection for feeding 3D data for a selected patient onto the computer. The computer is configured for receiving such user input 3D data, optionally by comprising a reader configured for reading images e.g. 2D and/or 3D and comprising software generating the 3D data from the read images. Advantageously the user input 3D data is added as 3D data set to the database system.
In an embodiment the 3D surface data generation means comprises a 3D surface sensor system for determining at least a part of the 3D data for at least the surface section of the minimally invasive surgery cavity and transmitting means for transmitting the determined for 3D data to the computer, optionally depiction system is configured for adding determined 3D data as 3D data set to the database system. Preferably the computer system is configured for adding determined 3D data received from the 3D surface sensor system as 3D data set to the database system.
In an embodiment the sensor system is adapted for determining at least a part of the 3D data for at least the surface section of the minimally invasive surgery cavity and the sensor system is further configured for generating at least one of pre-operative data and/or intra-operative data.
The transmitting means may be any means suitable for transmitting said 3D data, preferably in digital form. Suitable transmitting means includes USB key transmitting means, wire based transmitting means and/or wireless transmitting means, such as Bluetooth.
The surface sensor system may comprise any type of surface sensor suitable for determining 3D data of the surface section. In a preferred embodiment the 3D surface sensor system comprises at least one local reference sensor, preferably for spatial reference. Advantageously the at least one local reference sensor is adapted to be positioned on a patient and/or on a support (surgery table) for a patient.
In an embodiment there are a plurality of reference sensors, the reference sensors are preferably configured to communicate to locate a position of each other to thereby define an X.Y.Z dimensional space—e.g. as describes in US 2007/060098 or U.S. Pat. No. 6,631,271. In an embodiment the 3D surface sensor system is as described in “Spatial Data Estimation in Three Dimensional Distributed Wireless Sensor Networks”, by Karjee et al. Embedded Systems (ICES), 2014 International Conference 3-4 Jul. 2014, IEEE ISBN 978-1-4799-5025-6.
In an embodiment the 3D surface data generation means comprises the surface position data generation means for providing surface position data of the target area. The surface position data generation means advantageously comprises the at least one local reference sensor described above.
The 3D surface sensor system may advantageously comprise a 3D optical sensor, an acoustic sensor, a magnetic sensor, an electric sensor, an electromagnetic sensor or any combinations thereof.
In an embodiment the 3D surface sensor system comprises a surface tracking system as described in “Augmented reality in laparoscopic surgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20 (2011) 189-201.
The one or more sensors of the sensor system can in principle be positioned anywhere as long as it is capable of sensing at least a part of the surface section. Advantageously the one or more sensors of the sensor system is/are part of the depiction system. Where the one or more sensors of the sensor system is/are fixed to or integrated with one or more elements, this/these one or more elements advantageously form part of the depiction system.
In an embodiment at least one sensor is positioned on or is adapted to be positioned on a trocar and/or cannula, the trocar and/orcannula is preferably a part of the depiction system. In an embodiment at least one sensor is positioned on or is adapted to be positioned on a surgical instrument, preferably on a surgical tool of the surgical instrument for use in minimally invasive surgery, the surgical instrument or the surgical tool is preferably a part of the depiction system. In an embodiment at least one sensor is positioned on or is adapted to be positioned on an endoscope for use in minimally invasive surgery, the endoscope is preferably a part of the depiction system. In an embodiment at least one sensor is positioned on or is adapted to be positioned on a patient e.g. in the minimally invasive surgery cavity, such as on a surface thereof or external on the surface of the skin of a patient. In an embodiment at least one sensor is positioned on or is adapted to be positioned on a sensor instrument which can be at least partly inserted into the minimally invasive surgery cavity, the sensor instrument is preferably a part of the depiction system. In an embodiment at least one sensor is adapted to be positioned external to the minimally invasive surgery cavity e.g., to be manually handled or handled by a robot, preferably the external sensor (e.g. a scanning instrument—CT, NMR, MR, UV and/or IR scanner) and optionally the robot form part of the depiction system.
In an embodiment the 3D surface sensor system comprises a 3D optical sensor system comprising at least one optical source and at least one optical reader such as an image recorder comprising a 2D array of detectors, such as a camera, the 3D optical sensor system is preferably a binocular sensor system or a multiocular sensor system.
The optical source of the 3D optical sensor source may in principle comprise any kind of optical source, such as an illumination source, an invisible source—e.g. an IR source.
The optical source may be a coherent light source or an incoherent light source. Examples of optical sources includes a semiconductor light source, such as a laser diode and/or a VCSEL light source as well as any kind of laser sources including narrow bandwidth sources and broad band sources.
In an embodiment the optical source has a band width (full width at half maximum—FWHM) of up to about 50 nm, such as from 1 nm to about 40 nm. Preferably the narrow band width of the optical source is about 25 nm or less, such as about 10 nm or less.
In an embodiment the optical source is a broad band light source such as a supercontinuum light source e.g. spanning at least an octave within the bandwidth range from 400 nm to 2600 nm. Above 2600 nm light transmitted in a silica fiber will be strongly attenuated.
In an embodiment the optical source is configured for emitting at least one electromagnetic wavelength within the UV range of from about 10 nm to about 400 nm, such as from about 200 to about 400 nm.
In an embodiment the optical source is configured for emitting at least one electromagnetic wavelength within the visible range of from about 400 nm to about 700 nm, such as from about 500 to about 600 nm.
In an embodiment the optical source is configured for emitting at least one electromagnetic wavelength within the IR range of from about 700 nm to about 1 mm, such as from about 800 to about 2500 nm.
In an embodiment the optical source comprises at least one wavelength within the invisible range, such as the UV or the IR range.
Advantageously the optical source is tunable in wavelength and/or power.
The optical source is advantageously connected to or integrated in an endoscope.
In an embodiment the 3D optical sensor system comprise two optical sources, preferably laser sources of equal or different wavelength(s) and being emitted towards the surface in different light beam angles (determined as the angle of the centremost ray of the respective light beams).
In an embodiment the 3D optical sensor system is based on USING BEAM PHASE interference determination e.g. as described in US2004/0145746.
In an embodiment the 3D optical sensor system may for example comprise a phase optic element (e.g. a diffractive optic element (DOE)), a spatial light modulator, a multi-order diffractive lens, a holographic lens, a Fresnel lens and/or a computer regulated optical element. In an embodiment the DOE is as described in US 2013/0038836 e.g. as shown in
In an embodiment the 3D optical sensor system comprises two cameras e.g. as described in WO 2013/163391.
In an embodiment the at least one optical source is configured for emitting an optical tracking light pattern which pattern when impinged onto and reflected and/or scattered from the surface reveal the contour of the surface to thereby provide 3D data to be recorded by the optical reader. In this embodiment the 3D optical sensor system may advantageously be or comprise the apparatus described in US 2013/0296712 and/or EP 2 586 34.
In an embodiment at least one of the optical pattern emitting source and the recorder is positioned on an endoscope which advantageously forms part of the system.
In an embodiment the optical source is adapted for emitting a substantially stationary optical pattern onto at least a part of the surface section which pattern when impinged onto and reflected and/or scattered from the surface reveal the contour of the surface to thereby provide 3D data to be recorded by the optical reader and the optical pattern emitting source optionally being positioned on an endoscope which is adapted to be held substantially stationary during the minimally invasive surgery procedure, such as on an endoscope also comprising the recorder.
In the following the term “reflected and/or scattered from a surface” should be interpreted to include any interactions with the relevant surface which is readable by an optical recorder. The light in question may be visibly or invisible to the human eye e.g. including wavelengths from below 300 nm to above 2 μm.
Advantageously the depiction system preferably comprises both a stationary and a dynamic optical pattern source. The dynamic optical pattern source is preferably in form of a light source adapted for emitting said dynamic optical pattern onto at least a part of the surface section. The dynamic optical pattern is preferably dynamic by being mounted onto a surgical tool such that the reflected and/or scattered pattern (the pattern reflected and/or scattered from the surface section), which may be recorded by the recorder is changing in relation to movements of the surgical tool. Preferably the changes of the reflected and/or scattered pattern are correlated to the movements of the surgical tool. The correlated movement of the pattern may e.g. be provided as described in WO 15/124159 e.g. comprising the set comprising a surgical instrument wherein the surgical tool as described herein includes the body and the surgical tool of the surgical instrument of WO 15/124159. The correlated movement of the pattern may e.g. be provided as described in DK PA 2015 70483. The optical recorder may be mounted on the surgical tool and/or on an endoscope or at any other suitable means.
The optical source may be temporarily or permanently fixed to the surgical tool and the optical source preferably also form part of the position data generation means.
By simultaneously use of the a stationary and a dynamic optical pattern source both the 3D data—or at least some of the 3D data optionally supplemented with one or more 3D data set(s) from the 3D database system—and the real time position data can be generated simultaneously.
In an embodiment the depiction system comprises a dynamic optical pattern preferably dynamic by being mounted onto a surgical tool as described above and simultaneously the system comprises an optical recorder mounted to the surgical tool or to an endoscope for collecting both 3D data and position data and preferably orientation data.
In an embodiment the 3D surface sensor system comprises an acoustic sensor system comprising at least one sonic wave emitter and at least one sonic wave receiver. Preferably the acoustic sensor system comprises two or more sonic wave receivers, such as a 2D array of sonic wave receivers e.g. piezo sonic wave receivers.
Advantageously the acoustic sensor system comprises an ultrasound sensor. The 3D optical sensor system may e.g. be as described in any one of U.S. Pat. No. 4,694,434 of US 2014/0204702.
In an embodiment the 3D surface sensor system comprises a magnetic sensor system, such as a magnetic induction tomography sensor system and/or a magnetic resonance tomography sensor and/or an electromagnetic sensor system.
In an embodiment the 3D surface sensor system comprises an electric sensor system, such as a micro mechanical sensor system.
In an embodiment the 3D surface sensor system comprises an X-ray computed tomography sensor (CT scanning sensor).
The position data generation means may be any kind of position data generation means suitable for determining the real time spatial position of at least a part of the surgical tool. Preferably the part of the surgical tool includes at least the distal end of the tool.
In an embodiment the position data generation means for obtaining real time spatial position data of at least a part of the surgical tool comprises a position sensor system. The position sensor system preferably comprises a 3D optical sensor, an acoustic sensor, a magnetic sensor, an electric sensor, an electromagnetic sensor, an accelerometer, a gyroscope, a gravimeter an inertial navigation system a local positioning system or any combinations thereof.
Advantageously the position sensor system is part of the depiction system.
In an embodiment the position sensor system comprises an inertial navigation system. The inertial navigation system advantageously includes at least a computer (e.g. of the computer system) and a platform or module containing accelerometers, gyroscopes and/or other motion-sensing devices. The inertial navigation system may initially be provided with its position and velocity from another source (such as a human operator, a GPS satellite receiver, etc.), and thereafter computes its own updated position and velocity by integrating information received from the motion sensors. The advantage of an inertial navigation system is that it requires no external references in order to determine its position, orientation, or velocity once it has been initialized.
In an embodiment the position sensor system comprising a sensor element adapted to be or being physically connected to or integrated with the surgical tool.
In an embodiment the position sensor system comprises a magnetic motion capture systems comprising at least one sensor adapted to be or being physically connect to or integrated with the surgical tool to measure low-frequency magnetic fields generated by a transmitter source.
In an embodiment the position sensor system comprises a MEMS sensor magnetic motion capture systems adapted to be or being physically connect to or integrated with the surgical tool to measure return signal upon activation by a transponder.
In an embodiment the position sensor system comprises an acoustic sensor including at least one sensor mounted on integrated with the surgical tool the surgical tool for increase accuracy—e.g. for direction determination.
The one or more sensors of the position sensor system may advantageously be positioned as described above for the sensor(s) of the 3D sensor system optionally with an additional sensor mounted on or integrated with the surgical tool. Advantageously one or more sensors may be part of both the 3D sensor system and the position sensor system.
In an embodiment the position sensor system comprises at least one local reference sensor, preferably for spatial reference. The at least one local reference sensor is preferably adapted to be positioned on a patient, such as in the minimally invasive surgery cavity or on the outer skin of the patient and/or on a support (e.g. a surgery table) for a patient.
In an embodiment the position sensor system comprises a plurality of reference sensors—preferably also being a part of the 3D surface sensor system. The reference sensors are preferably configured to communicate to locate a position of each other to thereby define an X.Y.Z dimensional space—e.g. as describes in US 2007/060098 or U.S. Pat. No. 6,631,271. In an embodiment the position sensor system is based on the technology described in “Spatial Data Estimation in Three Dimensional Distributed Wireless Sensor Networks”, by Karjee et al. Embedded Systems (ICES), 2014 International Conference 3-4 July 2014, IEEE ISBN 978-1-4799-5025-6.
In an embodiment the at least one reference sensor provides the local positioning system, preferably the position sensor system comprises a plurality local reference sensors.
The position sensor system is advantageously configured for obtaining real time spatial position data of the surgical tool and for transmitting the obtained spatial position data to the computer system, wherein the real time spatial position data is on form of real time spatial position data in an X-Y-Z dimensional space.
In an embodiment the position sensor system comprises at least one distance sensor, such as a short range laser based sensor e.g. as provided by SICK AG, Germany.
A short range laser based distance sensor is operating by projecting a light beam spot onto a measurement object, e.g. using a laser diode. By means of an optical receiver, the reflection is mapped onto a light sensitive element (such as CMOS). Based on the position of the mapped light spot and the 3D surface data, the distance to the surface section can be determined.
Advantageously the real time spatial position data comprises position data correlated to the 3D data representing at least one surface section in 3D space, preferably the real time spatial position data comprises determining a distance from the surgical tool or the at least one part of the surgical tool to the surface section, preferably to the target area of the surface section.
In an embodiment the real time spatial position data comprises determining a distance e.g. using a short range laser based distance sensor from the surgical tool or at least one part of the surgical tool—preferably its distal end—and to a point of the surface section correlated with an orientation direction of the tool, where the orientation direction of the tool is determined from an access opening into the minimally invasive surgery cavity through which the tool is inserted and the farthest extension of the tool into the cavity. This orientation direction is also referred to as the longitudinal vector direction of the surgical tool and/or where the surgical tool is not bended or bendable—the longitudinal direction of the surgical tool. Where the surgical tool is bended the longitudinal direction is determined as the direction of the distal part of the surgical tool from the bend to the most distal part of the tool.
In an embodiment the real time spatial position data comprises determining a distance from the surgical tool or at least one part of the surgical tool—preferably its distal end—and to a point of the surface section in distal direction to the surgical tool.
Advantageously the 3D surface sensor system and the position sensor system are at least partly integrated to form a combined 3D surface and position sensor system. Preferably this 3D surface and position sensor system is a part of the depiction system.
In an embodiment the computer system is in data connection with the 3D surface data generation means and is configured for acquire the 3D data representing at least one surface section in 3D space of a minimally invasive surgery cavity from the 3D surface data generation means. Preferably the computer system is configured for receiving instruction via a user interface to acquire the 3D data and upon receiving such instructions to acquire the 3D data.
In an embodiment the computer system—for each user procedure—is configured to acquire the 3D data representing at least one surface section in 3D space of a minimally invasive surgery cavity in one single time data package. In this embodiment the acquired 3D data representing at least one surface section in 3D space of a minimally invasive surgery cavity may be used in the whole minimally invasive surgery procedure, and the 3D surface data generation means is advantageously controlled by the computer system to acquire the 3D data representing at least one surface section in 3D space of a minimally invasive surgery cavity only once in one data package.
In another embodiment the computer system—for each user procedure—is configured for acquiring the 3D data by consecutively acquiring 3D data packages representing at least one surface section in 3D space of a minimally invasive surgery. The consecutively acquired 3D data packages preferably comprises timely updated 3D data and/or 3D data representing at least one additional surface section in 3D space for each of the consecutive acquired 3D data packages. Advantageously the operation of the 3D surface data generation means is controlled by the computer system.
In an embodiment the computer system is in data communication with the position data generation means and is configured for in real time acquire the obtained real time spatial position data of at least a part of the surgical tool. Preferably the computer system is configured for receiving instruction via a user interface to acquire the spatial position data and upon receiving such instructions to acquire the spatial position data.
In an embodiment the computer system is configured for controlling the 3D surface data generation means and the position data generation means. Preferably the computer is configured for receiving instruction via a user interface to conduct a user process and based on the instruction to acquire the required data from the 3D surface data generation means and the position data generation means and for conducting the user procedure until a termination signal is transmitted to the computer.
Advantageously the system further comprises orientation data generation means for obtaining real time spatial orientation data of at least a part of the surgical tool and for transmitting the obtained spatial orientation data to the computer system. The computer system is advantageously programmed for
Advantageously the computer system is configured to calculating depiction data representing the depiction of the real time relative spatial position(s) and the real time spatial orientation(s) of the surgical tool onto at least a portion of the surface contour of the surface section of the minimally invasive surgery cavity comprises depiction data representing an associated depiction of the real time relative spatial position(s) and the real time spatial orientation(s) of the surgical tool onto at least a portion of the surface contour.
Thereby an effective depiction with a concentrated amount of information can be visualised for the user.
In an embodiment the associated depiction of the real time relative spatial position(s) and the real time spatial orientation(s) of the surgical tool onto at least a portion of the surface contour comprises a depiction of the real time relative spatial position(s) onto the surface contour in a direction coordinated with the real time spatial orientation(s) of the surgical tool. The resulting depiction will thereby be a dynamically changing depiction and will thereby show even smaller movements of the surgical tool including tilting movements which will bring very useful information to the user.
Advantageously the orientation data generation means comprises an orientation sensor connected to or integrated with the surgical tool. In an embodiment the surgical tool is part of the depiction system and comprises integrated at least one sensor.
In an embodiment the orientation data generation means comprises an orientation sensor connected to or integrated with the surgical tool. The orientation sensor preferably comprises at least one distance sensor, such as a short range laser based sensor described above. The at least one direction sensor preferably being mounted to emit laser light in distal direction relative to the surgical tool, preferably oriented substantially parallel to the longitudinal direction of the surgical tool.
In an embodiment the position data generation means and the orientation data generation means are at least partly integrated to a combined position data and an orientation data generation means. In practice it will be simpler to have a combined position data and an orientation data generation means.
In this embodiment the position data/position data generation means also comprises orientation data/orientation data generation means.
Preferably the depiction data representing a depiction of the real time relative spatial position(s) of the surgical tool onto the determined 3D surface contour of the surface section of the minimally invasive surgery cavity comprises depiction data encoding a depiction of a dynamic pattern representation, a dynamic scaling of colours representation, a dynamic schematic representation, a dynamic graphical representation and/or a dynamic augmented reality representation.
Advantageously the encoded depiction comprises a non-image-accurate depiction of the surgical tool.
Whereas prior art systems heretofore have been focused on generating 3D vision of a minimally invasive surgery area or a surgical instrument which is an imitation of a real life vision of the minimally invasive surgery area or a surgical instrument the system of the present invention provides a depiction comprising a correlated projection of the real time position and movements and preferably real time orientation of the surgical tool rather than imaging the surgical tool as such. By providing the depiction of the real time relative spatial position(s) of the surgical tool onto the determined 3D surface contour of the surface section of the minimally invasive surgery cavity in form of depiction data encoding a depiction of a dynamic pattern representation, a dynamic scaling of colours representation, a dynamic schematic representation, a dynamic graphical representation and/or a dynamic augmented reality representation, wherein the encoded depiction does not comprise a non-image-accurate depiction of the surgical tool, the resulting depiction can comprise a very high concentration of useful information to the user. For easy decoding of the depiction the user may be trained e.g. as described further later on. Advantageously the depiction also includes sound as exemplified below.
In an embodiment the dynamic scaling of colours comprises a dynamic scaling of shades of one or more colours, such as hue colours.
The term shades are herein used to designate a gradation of a colour with more or less black and/or brightness.
In an embodiment the depiction data comprises data encoding a depiction of a dynamic pattern representation, wherein the dynamic pattern representation comprises a depiction of a virtual pattern resembling an emitted light pattern impinged onto the determined 3D surface contour, wherein the virtual pattern preferably comprises arch shaped and/or ring shaped lines and/or a plurality of angled lines.
Advantageously the dynamic pattern representation comprises a depiction of the virtual pattern onto the determined 3D surface contour, such that the depiction comprises a dynamic modification of the virtual pattern wherein the dynamic modification is correlated to the determined real time spatially position(s) and thereby movements of the surgical tool.
The virtual pattern preferably resembling a light pattern comprising at least spatially dispersed light beam fractions, such as an angular light path surrounding a light dot or a grid of lines, e.g. a crosshatched pattern optionally comprising substantially parallel lines when emitted to a planar surface, and/or one or more angular, such as rectangular shapes e.g. square shaped, e.g. in an overlapping configuration, in a side by side configuration or concentrically arranged.
In an embodiment the dynamic pattern representation comprises a depiction corresponding to the light pattern reflected and/or scattered from the surface of the surgery site when using the systems and methods disclosed in WO15124159 and/or in co-pending patent application DK PA 2015 70483.
In an embodiment the depiction comprises a depiction of a dynamic scaling of colours representation, wherein the dynamic scaling of colours representation comprises a visual coloured representation of the determined real time spatially position(s) of the surgical tool relative to the determined 3D surface contour wherein the colouring is dynamically modified in correlation to changes of the spatial position and optional orientation caused by the movements of the surgical tool relative to the determined 3D surface contour for example such that the shorter distance between the surgical instrument and the target area the more intensive red and/or less intensive green.
The depiction of a dynamic scaling of colours representation has shown to be very simple to decoding for most users and can comprise a large amount of useful information. Advantageously the depiction systems comprise means (a button or wireless setting means) for setting the colour scaling and brightness. For some users the colour scaling is not sufficient e.g. due to colour blindness and in such situation other depiction are preferred e.g. in combination with the colour scaling representation.
In an embodiment the visual coloured representation comprises a 2D graduated shading.
In an embodiment the depiction comprises a dynamic schematic representation, wherein the dynamic schematic representation comprises a diagrammatic representation of the determined real time spatially position(s) of the surgical tool relative to the determined 3D surface contour wherein the diagrammatic representation is dynamically modified in correlation to changes of the spatial position and optional orientation caused by the movements of the surgical tool relative to the determined 3D surface contour.
In an embodiment the depiction comprises a depiction of a dynamic graphical representation, wherein the dynamic graphical representation comprises a graphical representation of the determined real time spatially position(s) of the surgical tool relative to the determined 3D surface contour wherein the graphical representation is dynamically modified in correlation to changes of the spatial position and optional orientation caused by the movements of the surgical tool relative to the determined 3D surface contour.
In a preferred embodiment the depiction comprises a depiction of a dynamic augmented reality representation, wherein the dynamic augmented reality representation comprises an augmented reality representation of the determined real time spatially position(s) of the surgical tool relative to the determined 3D surface contour wherein the augmented reality representation is dynamically modified in correlation to changes of the spatial position and optional orientation caused by the movements of the surgical tool relative to the determined 3D surface contour.
In an embodiment the augmented reality is a spatial augmented reality (SAR) wherein the display unit comprises one or more digital projectors configured to display the depiction upon receipt of the calculated depiction data e.g. onto a physical object(s), such as onto the patient, such as onto an artificial patient or a head-up display.
Advantageously the depiction comprises a sound depiction, such as a beeping sound where the tone and/or the beep-rate is correlated to the relative distance between the surgical tool and the surface section. The beeping sound may for example intensify as the surgical tool is approaching the surface section e.g. to a point of the surface section distal to the surgical tool in its longitudinal direction.
In an embodiment the depiction comprises a vibration depiction, such as a vibration of the surgical tool, such as of a proximal part of the surgical tool adapted to be sensed by a surgeon holding or being mounted with the surgical tool. The vibration may for example be activated in case the surgical tool is dangerously close to a critical structure of the surface section of the minimally invasive surgery cavity.
In an embodiment the calculated depiction data is configured for being displaced on the display unit, wherein the display unit is selected from a 2D screen, a 3D screen, a projector system (such as an augmented projector system), head-up display, a wearable display unit, such as a head mounted display (e.g. goggles) or any combinations thereof.
In an embodiment the calculated depiction data comprises pixel data for a pixel based display unit, such as a screen, the pixel data preferably comprises red-blue-green pixel data, preferable for displaying high colour or true colour comprising at least 16 bits per pixel (bpp), such as at least 20 bpp.
The pixel data of the depiction data representing a depiction of the real time spatial position(s) of the surgical tool onto the determined 3D surface contour of the surface section of the minimally invasive surgery cavity is preferably dynamically modified in correlation to changes of the spatial position and optional orientation caused by the movements of the surgical tool relative to the determined 3D surface contour.
To obtain a high quality depiction the calculated depiction data advantageously comprises pixel data for a 2D display, such as a 2D display of at least about 1000 pixels, such as at least 10.000 pixels optionally comprising sub-pixels.
In an embodiment the display unit is part of the depiction system, the display unit is preferably selected from a 2D screen, a 3D screen, a projector system (such as an augmented projector system), head-up display, a wearable display unit, such as a head mounted display (e.g. goggles) or any combinations thereof.
To increase the visual perception of the user it is in an embodiment desired that the depiction system comprises a real imaging system configured for generating real imaging data for a real imaging of the at least one surface section of the minimally invasive surgery cavity. The real imaging system is a 2D real imaging system, a 3D real imaging system, a virtual reality real imaging system, an augmented reality real imaging system or any combination thereof.
The term “real imaging” is herein used to mean an imaging where the images show or images the surface section as it is in real life.
The real imaging system may for example be an endoscopic system as it is well known today, e.g. an endoscope for inserting into the minimally invasive surgery cavity and comprising an illuminator for illumination the surface target area and a camera for requiring real images of the target area.
In an embodiment the real imaging system is an augmented reality real imaging system e.g. as describe in“Augmented reality in laparoscopic surgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20 (2011) 189-201 and “An effective visualization technique for depth perception in augmented reality-based surgical navigation” by Choi Hyunseok et al. The international journal of medical robotics and computer assisted surgery, 2015 May 5. doi: 10.1002/rcs.1657.
Advantageously the real imaging system comprises an image recorder (camera) for acquiring at least one image of the at least one surface of section of the minimally invasive surgery cavity. The image recorder preferably being a part of an endoscope, the image recorder preferably being a videoscope preferably acquiring real time images of the at least one surface section of the minimally invasive surgery cavity.
The image recorder is for example a 2D recorder comprising a 2D array of detectors in form of pixel detectors, preferably the image recorder comprises at least 1000 pixels, such as at least 10.000 pixels, preferably the image recorder is a mega pixel recorder comprising at least 1 mega pixels.
In an embodiment the image recorder is a 3D recorder comprising a 3D array of detectors in form of pixel detectors, preferably the image recorder comprises at least 1000 pixels, such as at least 10.000 pixels, preferably the image recorder is a mega pixel recorder comprising at least 1 mega pixels.
In an embodiment the image recorder or preferably the whole real imaging system constitutes also a part of the 3D surface data generation means for providing the computer system with the three-dimensional (3D) data representing the at least one surface section in 3D space of the minimally invasive surgery cavity.
Preferably the depiction system is configured for transmitting the real imaging data to the display unit for providing a real imaging of the at least one surface section of the minimally invasive surgery cavity, the real imaging preferably comprises displaying the acquired image(s).
By displaying both the real imaging of the at least one surface section of the minimally invasive surgery cavity and the depiction of the real time relative spatial position(s) of the surgical tool onto the surface contour of the surface section of the minimally invasive surgery cavity the visual perception of the user is highly increased. Advantageously depiction system is configured for displaying the real imaging of the at least one surface section of the minimally invasive surgery cavity and the real time correlated depiction of movements of the surgical tool by a common display unit.
Advantageously the depiction system is configured for displaying the real imaging of the at least one surface section of the minimally invasive surgery cavity and onto the real imaging of the at least one surface section of the minimally invasive surgery cavity to display the depiction of the real time relative spatial position(s) of the surgical tool onto the surface contour of the surface section of the minimally invasive surgery cavity.
Preferably the depiction of the real time relative spatial position(s) of the surgical tool onto the surface contour of the surface section of the minimally invasive surgery cavity is projected onto the real imaging of the at least one surface section of the minimally invasive surgery cavity, the depiction of the real time relative spatial position(s) of the surgical tool onto the surface contour of the surface section of the minimally invasive surgery cavity is preferably at least partly transparent for the real imaging of the at least one surface section of the minimally invasive surgery cavity. Thereby the user can on the same display se both the real imaging of the at least one surface section of the minimally invasive surgery cavity and onto the real imaging of the at least one surface section of the minimally invasive surgery cavity to display the depiction of the real time relative spatial position(s) of the surgical tool onto the surface contour of the surface section of the minimally invasive surgery cavity in a correlated way, which further increase the visual perception.
In an embodiment the depiction system comprises a robot controller for controlling a robot which preferably form part of the depiction system. The robot and the robot controller may in an embodiment be as described in US 2009/0248041. The robot controller is in data connection with or is integrated with the computer system. In the following the robot controller is describe as being in data communication with the computer system but it should be interpreted such that the computer controller as well can be an integrated part of the computer system.
The robot controller being configured for receiving at least a part of the obtained or generated data preferably comprising said 3D surface data and said real time spatial position data and preferably said real time orientation data. Optionally the robot controller is also configured for receiving derived data—i.e. data derived from the 3D surface data and said real time spatial position data and preferably said real time orientation data—such as the depiction data. The robot controller is preferably configured for controlling the robot for handling the surgical tool for performing a minimally invasive surgery procedure.
In an embodiment the robot controller is configured for acquiring the required data from the computer system respective data from said computer system.
Advantageously the robot controller is configured for receiving instructions from a supervisor, such as an assisting surgeon and/or a robot operator. The controller is preferably configured for receiving instructions from the supervisor by data input via a digital user interface and/or by oral instruction.
The robot controller is preferably configured to modify movements of said surgical tool in response to the supervisor instructions. The supervisor instruction may for example instruct the robot to move the surgical tool a cm to the right or a few mm distally in longitudinal direction of the surgical tool or etc.
In an embodiment the depiction system is configured for simultaneously, via the robot controller, controlling the robot for handling the surgical tool for performing a minimally invasive surgery procedure and for transmitting said depiction data to the display unit. The depiction will in this embodiment comprise a real time correlated depiction of said movements of said surgical tool by said robot. The supervisor may keep the robot under observation during its performing of the minimally invasive surgery via the depiction and thereby control that the robot is operation sufficiently accurate or as explained, the supervisor may correct the robot by feeding instructions to the robot controller.
The computer system preferably comprises a memory and is advantageously configured for storing performance data sets, where each performance data set comprises performance data associated with a minimally invasive procedure, such as for minimally invasive procedure that has been performed using the depiction system. The performance data preferably comprises at least the position data for the minimally invasive procedure, and preferably at least one of the 3D data, the orientation data, the depiction data for the minimally invasive procedure or any combination thereof. Preferably the performance data set comprises all the data acquired during a minimally invasive surgery procedure. In an embodiment the performance data set further comprises calculated data including the depiction date.
In an embodiment the performance data comprises at least one of the position data, the 3D data, the orientation data or the depiction data for the minimally invasive procedure.
In an embodiment the computer system is programmed to analyse the respective performance data set and to transmit a feedback evaluation(s) of the respective minimally invasive procedures to a display means.
In an embodiment the computer system is programmed to analyse the respective performance data set and preferably the computer system is configured to transmit a feedback evaluation(s) of the respective minimally invasive procedures to a display means, such as the display unit and/or a printer.
Thereby a user—i.e. a surgeon or a person under training for becoming a surgeon—can in a very simple way receive feedback and he can evaluate his improvement. Advantageously the computer system is configured to transmit a feedback evaluation(s) upon a request. Each performance data set preferably has a unique code such that the respective performance data sets can be retrieved for displaying of the depiction of the real time relative spatial position(s) of the surgical tool e.g. for the surgeon or for training purpose of other surgeons.
In an embodiment the computer system is programmed to determine an improved performance data set relative to a selected performance set, and to transmit at least a part of the data, such as depiction data of the improved performance data set to a display means, such as the display unit and/or a printer.
Preferably the computer is programmed to compare the improved performance data set relative to the selected performance set and the selected performance set to determine data differences and to transmit the data differences to a display means, such as the display unit and/or a printer.
Thereby the user can compare his performance with a superior performance data set e.g. a calculated performance data set and he can retrieve information about what can be improved, where he uses superfluous movements etc.
In an embodiment the computer system is programmed to determine difference in performance data, categorizing the data and store the categorized data for machine learning purposes.
In an embodiment the computer system is programmed to transmit selected data of the one or more performance data set to a display means, such as the display unit and/or a printer, preferably upon request from a user.
Thereby the respective performance data sets can be retrieved for displaying of the depiction of the real time relative spatial position(s) of the surgical tool e.g. for use in education.
In an embodiment the depiction system is configured for operating in a training mode. The surgical tool is a training tool, which may or may not resemble a real surgical tool.
The depiction system when operating in its training mode is configured for determine training depiction data representing a training depiction of the real time relative spatial position(s) of the training tool onto at least a portion of the surface contour of the surface section of the minimally invasive surgery cavity, and for transmitting the training depiction data to the display unit for real time correlated depiction of movements of the training tool.
When the depiction system is in its training mode the minimally invasive surgery cavity surface section may for example be an artificial surface section and in this embodiment the training tool may advantageously be an artificial tool.
In an embodiment the minimally invasive surgery cavity surface section when the depiction system is in its training mode is an actual minimally invasive surgery cavity. In this embodiment the training tool is a real surgical tool. In principle the surgeons is constantly in training for improving their technique in order to perform a minimally invasive surgery as optimal as possibly—e.g. using the least movements of the surgical tool possibly and/or performing the minimally invasive surgery as fast as possible.
In an embodiment the position data generation means being configured for obtaining real time spatial position data of at least a part of the training tool and for transmitting the obtained spatial position data to the computer system, the computer system being programmed for
The computer system is advantageously configured for transmitting the training depiction data and associated performance depiction data of stored performance data set for the same or a corresponding minimally invasive surgery cavity to the display unit for real time correlated depiction of movements of the training tool. Preferably the transmission of the training depiction data and associated performance depiction data are timely associated.
Thereby a user during training can benchmark against an earlier performed minimally invasive surgery procedures or calculated procedures or his own earlier procedures.
In an embodiment the computer system is configured for transmitting the training depiction data and associated performance depiction data of stored performance data set for the same or a corresponding minimally invasive surgery cavity for performing a benchmarking of the performance depiction data relative to said training depiction data.
In an embodiment the performance depiction data set is transmitted to the display unit in a speed of about real-time speed.
In an embodiment the performance depiction data set is transmitted to the display unit in a speed which is less than real-time speed.
Thereby a person under training can perform the minimally invasive surgery procedure at a slower pace for training purpose and he can step-by-step increase the pace to reach the real time speed or even faster if desired for training purpose.
In an embodiment the computer system is configured for receiving supervisor input e.g. from a user or a training database, to determine supervisor depiction data based on the supervisor input and transmitting the supervisor depiction data to the display unit, preferably the computer is configured for acquiring the supervisor input from a training database. The supervisor input may be input representing selected positions and movements of the training tool.
In an embodiment the superviser is a robot in data communication with the computer system.
Advantageously the computer system is configured for transmitting the training depiction data and the associated performance depiction data and/or the supervisor depiction data to the display unit to be displayed separately, such as in a side by side configuration. Thereby the person under training, such as a surgeon aiming to improve his skill can be trained to position the training tool in a selected position or moving the training tool in a select way as instructed by the supervisor input. The person under training can try to move the training tool such that the training depiction follows associated performance and/or the supervisor depiction.
In an embodiment the supervisor is another surgeon observing and advising a surgeon during a minimally invasive surgery procedure. In an embodiment there are for example two surgeons operating together, where one is handling the surgical tool and the other one is acting as supervisor making suggestions for positioning and movements of the surgical tool and visa verse. It should be understood that the display unit may be in form of two or more sub unit.
All features of the inventions including ranges and preferred ranges can be combined in various ways within the scope of the invention, unless there are specific reasons not to combine such features.
Preferred embodiments of the invention will be further described with reference to the drawings.
The figures are schematic and are not drawn to scale and may be simplified for clarity. Throughout, the same reference numerals are used for identical or corresponding parts.
The depiction system illustrated in
The computer system 1 is programmed for
The display unit can be as described above.
The depiction system illustrated in
The surgical instrument shown in
The surgical instrument has a handle portion 22 and a body tool portion 23 with a surgical tool portion 24 in the present case forceps. The surgical tool portion 24 and the part of the body portion 23 adapted to be inserted into the minimally invasive surgery cavity is referred to as the surgical tool. In other words the surgical instrument 21 comprises the surgical tool 23, 24 and the handle 22. The body portion 23 interconnect the handle portion 22 which is arranged at the proximal end of the surgical instrument and the surgical tool portion 24, which is arranged at the distal end of the surgical instrument. The body portion 23 is arranged in the distal/proximal direction, which is also referred to as the longitudinal direction of the surgical tool 23, 24.
In another embodiment the surgical tool portion 24 may be another surgical tool portion e.g. a grasper, a suture grasper, a stapler, a dissector, scissors, a suction instrument, a clamp instrument, an electrode, a curette, ablators, scalpels, a needle holder, a biopsy and retractor instrument or a combination thereof.
The surgeon operate the surgical instrument 21 by holding the handle portion 22 and can in this way control the surgical tool and by pressing or manipulating the handle portion the forceps can be controlled.
The surgical instrument 21 further comprises a first sensor 28a, such a sensor as described above, mounted to the surgical tool and a second sensor 28b mounted at the handle. The sensor may be linked to the not shown data collecting system of the computer system e.g. by wire (for example optical fiber(s) or wireless e.g. blue tooth. The two sensors may provide both real time position data as well as real time orientation data.
In an alternative embodiment the surgical instrument may comprise a pattern generating arrangement. The surgical tool with a handle may for example be in form of a surgical instrument assembly as described in WO15124159. The reflected/scattered pattern may thus be collected by a reader and transmitted to a data collecting system of the computer system to provide a part of the 3D surface data, which may preferably also be real time 3D data and the real time position data and preferably also real time orientation data.
The patient 36 is positioned on an operation table 33 with his back 34 against the table 33 and his front 35 upwards. The minimally invasive surgery cavity is blown up by injecting a gas into the cavity through a not shown incision and the surface is indicated with reference 32. As illustrated the minimally invasive surgery cavity surface 32 may be very uneven and with large curved projections and recesses.
A number of sensors 38a are positioned onto the operation table. Only 4 sensors 38a are shown, but preferably at least 4 sensors 38a are positioned in a rectangular configuration on the operation to define an X-Y-Z plan and optionally with additional sensors.
Further a number of sensors 38b are mounted to the patient e.g. within the minimally invasive surgery cavity 31 as shown.
The sensors 38a, 38b may e.g. be as described above.
An example of a real time correlated depiction of movements of a surgical tool is illustrated in
Another example of a real time correlated depiction of movements of a surgical too is illustrated in
A further example of a real time correlated depiction of movements of a surgical tool is illustrated in
A variation of the depiction of
The time correlated depiction of movements of a surgical tool illustrated in
The time correlated depiction of movements of a surgical tool illustrated in
As it can be understood the display and the depiction may take many different forms.
The depiction system of
The surgical tool may have a handle for example be in form of a surgical instrument assembly as described in WO15124159.
The surgical tool is inserted through a not shown incision through the skin layer 106 of a patient and into the minimally invasive surgery cavity 100a. The light pattern is emitted towards the relevant surface section 100b and the light pattern 109 is impinging on and a part is reflected and/or scattered from the surface section 100b of the minimally invasive surgery cavity 100a. The endoscope comprises an optical recorder for recording the reflected and/or scattered light pattern 109, and the collected data which advantageously includes 3D surface data (in real time), real time position data and real time orientation date is transmitted by wire or wireless to the data collection system 102. In a variation thereof the optical recorder 108 is not a part of an endoscope but inserted through the skin layer 106 at another place than the site of the endoscope entry. In another variation the optical recorder 108 is fixed to or integrated with the surgical tool.
The depiction system illustrated in
In the illustrated embodiment the depiction system comprises an endoscope 118 and a surgical instrument with a handle 117a and a surgical tool 117. The surgical tool may further emit a light pattern as described in
The endoscope 118 comprises a recorder for recording real images of the surface section and the surgical tool. The real images are collected in a secondary data collection system 112b of the computer system and transmitted to the computer from where they are transmitted for displaying on the screen 113 such that the real images 115 are timely associated with the depiction 119.
In the shown
The computer has information about the position of this target area T e.g. via the images recorded by the recorder 118 of by other means. The respective sensors S1 and S2 are advantageously distance sensors configured for determining the respective distances D1 and D2 to the target area T and preferably also the respective position of the sensors S1, S2 relative to the target area to thereby determine the orientation of the surgical tool 117. Optionally, the longitudinal direction of the surgical tool 117 is marked/depicted by a dot at target area T.
The real images 115 and the depiction are displayed on the display 113 besides each other. The target area T may also be shown on the real image 115. The depiction 119 is a graphical depiction showing the distance D1 between sensor S1 and the target area T as well as the distance D2 between sensor S2 and the target area T. e.g. by a bar indicator 199a that is moved in horizontal direction.
The depiction system illustrated in
The endoscope 138 comprises an arrangement for emitting a light pattern and a recorder for recording real images of the surface section and the surgical tool 137. The real images are collected in the data collection system 132 of the computer system and transmitted to the computer 131 from where they are transmitted for displaying on the screen 133 such that the real images 135 are timely associated with the depiction 139.
In the shown
The depiction system illustrated in
The depiction system illustrated in
The depiction system further comprises a supervisor control unit 154a in communication with or comprised in its computer system. The supervisor control unit comprises a digital user (supervisor) interface and/or a sound recorder for recording supervisor input, such as supervisor instructions. The supervisor input data are transmitted via the computer 151 to the display 153 for being displayed as a supervisor depiction 154b.
The depiction system illustrated in
In the illustrated embodiment the data generation means comprises an endoscope 168 comprising an arrangement for emitting a light pattern and a recorder for recording reflected/scattered light from the pattern and optionally for recording real images of the surface section and the surgical tool 167. In a variation of the shown embodiment the arrangement for emitting a light pattern and a recorder for recording reflected/scattered light from the pattern is not a part of an endoscope 168 but inserted through the skin layer 166 at another place than the site of the endoscope entry. The surgical tool 167 of the depiction system is also configured for emitting a light pattern.
The surgical tool with a handle may for example be in form of a surgical instrument assembly as described in WO15124159. The handle and/or the whole surgical tool may for example be a part of a robotic arm.
The endoscope 168 is inserted through an incision in the skin layer 166 of the patient. The endoscope is emitting a stationary pattern such as a crosshatched pattern 168a, which is impinging onto and at least partly reflected/scattered from the surface section of the cavity, thereby revealing the surface contour of the surface section which is recorded both in form of 3D surface data and preferably also in form of real images by the recorder of the endoscope 168.
The surgical tool 167 is inserted through another incision (or same incision) through the skin layer 166 of the patient and into the minimally invasive surgery cavity. The light pattern is emitted towards the relevant surface section and the light pattern 167a is impinging on and at least partly reflected and/or scattered from the surface section of the minimally invasive surgery cavity. As the surgical tool 167 is moved the pattern 167a emitted from the surgical tool 167 becomes a dynamic pattern 167a upon the stationary pattern 168a from the endoscope 168. As the reflected and/or scattered light is recorded by the recorder of the endoscope 168 large amounts of 3D surface data, real time position data as well as real time orientation data is obtained and transmitted to the data collection system 162. The movements of the dynamic pattern may be for example be related to the stationary pattern and the relevant 3D, position and orientation data may be determined using trigonometric calculation methods.
The depiction system further comprises a robot controller 160. The computer 161 is in data connection with the robot controller 160 for transmitting at lease a part of the collected data including 3D surface data, real time spatial position data and real time orientation data to the robot controller 160. The robot controller 160 is configured for controlling a not shown robot for handling the surgical tool for performing a minimally invasive surgery procedure and the depiction system is configured for displaying the depiction, which comprises a real time correlated depiction of movements of the surgical tool by the robot. Thereby a supervisor e.g. an operator can keep the robot under observation during its performing of the minimally invasive surgery via the displayed depiction and thereby control that the robot is operation sufficiently accurate or as explained, the supervisor may correct the robot by feeding instructions to the robot controller e.g. via a supervisor control unit as shown in
The figures are schematic and are not drawn to scale and may be simplified for clarity. Throughout, the same reference numerals are used for identical or corresponding parts.
This is a continuation of International Application PCT/DK2016/050180, with an international filing date of Jun. 13, 2016 and claiming priority from PA 2015 70642 DK of Oct. 9, 2015. International Application PCT/DK2016/050180 is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3919541 | Chao | Nov 1975 | A |
| 4694434 | von Ramm et al. | Sep 1987 | A |
| 4877016 | Kantor et al. | Oct 1989 | A |
| 4887222 | Miyake et al. | Dec 1989 | A |
| 5457439 | Kuhn | Oct 1995 | A |
| 5588949 | Taylor et al. | Dec 1996 | A |
| 5651783 | Reynard | Jul 1997 | A |
| 5710870 | Ohm et al. | Jan 1998 | A |
| 5725523 | Mueller | Mar 1998 | A |
| 5747953 | Philipp | May 1998 | A |
| 5754717 | Esch | May 1998 | A |
| 5769791 | Benaron et al. | Jun 1998 | A |
| 5836869 | Kudo et al. | Nov 1998 | A |
| 5888194 | Utsumi et al. | Mar 1999 | A |
| 5933223 | Flock et al. | Aug 1999 | A |
| 5951142 | Wang et al. | Sep 1999 | A |
| 5976077 | Wittens et al. | Nov 1999 | A |
| 5989181 | Duetting et al. | Nov 1999 | A |
| 6006127 | Van Der Brug et al. | Dec 1999 | A |
| 6017354 | Culp et al. | Jan 2000 | A |
| 6036636 | Motoki et al. | Mar 2000 | A |
| 6165184 | Verdura et al. | Dec 2000 | A |
| 6361530 | Mersch | Mar 2002 | B1 |
| 6387044 | Tachibana et al. | May 2002 | B1 |
| 6471638 | Chang et al. | Oct 2002 | B1 |
| 6494827 | Matsumoto et al. | Dec 2002 | B1 |
| 6522806 | James, IV et al. | Feb 2003 | B1 |
| 6527704 | Chang et al. | Mar 2003 | B1 |
| 6537290 | Adams et al. | Mar 2003 | B2 |
| 6549288 | Migdal et al. | Apr 2003 | B1 |
| 6631271 | Logan | Oct 2003 | B1 |
| 6659943 | Watanabe et al. | Dec 2003 | B2 |
| 6741883 | Gildenberg | May 2004 | B2 |
| 6791601 | Chang et al. | Sep 2004 | B1 |
| 6810184 | Skutnik | Oct 2004 | B2 |
| 6811546 | Callas et al. | Nov 2004 | B1 |
| 6908428 | Aizenfeld et al. | Jun 2005 | B2 |
| 6945981 | Donofrio et al. | Sep 2005 | B2 |
| 7037314 | Armstrong | May 2006 | B2 |
| 7049594 | Wu et al. | May 2006 | B2 |
| 7063695 | Nield et al. | Jun 2006 | B2 |
| 7113675 | Nield et al. | Sep 2006 | B2 |
| 7211044 | Mast et al. | May 2007 | B2 |
| 7422327 | Smith | Sep 2008 | B2 |
| 7464846 | Shelton, IV et al. | Dec 2008 | B2 |
| 7505808 | Anderson et al. | Mar 2009 | B2 |
| 7725214 | Diolaiti | May 2010 | B2 |
| 7775969 | Teichmann | Aug 2010 | B2 |
| 7784947 | Perez et al. | Aug 2010 | B2 |
| 7843558 | Furman | Nov 2010 | B2 |
| 7845537 | Shelton, IV et al. | Dec 2010 | B2 |
| 7876942 | Gilboa | Jan 2011 | B2 |
| 7912532 | Schmidt et al. | Mar 2011 | B2 |
| 7927272 | Bayer et al. | Apr 2011 | B2 |
| 7976459 | Laser | Jul 2011 | B2 |
| 8064819 | Ingrassia et al. | Nov 2011 | B2 |
| 8086008 | Coste-Maniere et al. | Dec 2011 | B2 |
| 8162826 | Pecherer et al. | Apr 2012 | B2 |
| 8165351 | Bendall | Apr 2012 | B2 |
| 8182422 | Bayer et al. | May 2012 | B2 |
| 8211044 | Liebowitz | Jul 2012 | B2 |
| 8242390 | Prest et al. | Aug 2012 | B2 |
| 8242398 | Young et al. | Aug 2012 | B2 |
| 8340379 | Razzaque et al. | Dec 2012 | B2 |
| 8397335 | Gordin et al. | Mar 2013 | B2 |
| 8403843 | Bruto Da Costa | Mar 2013 | B2 |
| 8409076 | Pang et al. | Apr 2013 | B2 |
| 8443007 | Kindig et al. | May 2013 | B1 |
| 8480566 | Farr | Jul 2013 | B2 |
| 8512368 | Sato et al. | Aug 2013 | B2 |
| 8525059 | Berger et al. | Sep 2013 | B2 |
| 8527033 | Williams et al. | Sep 2013 | B1 |
| 8531511 | Katakura | Sep 2013 | B2 |
| 8554307 | Razzaque et al. | Oct 2013 | B2 |
| 8568304 | Vayser et al. | Oct 2013 | B2 |
| 8632535 | Shelton, IV et al. | Jan 2014 | B2 |
| 8657809 | Schoepp | Feb 2014 | B2 |
| 8670816 | Green et al. | Mar 2014 | B2 |
| 8672207 | Shelton, IV et al. | Mar 2014 | B2 |
| 8708211 | Zemlok et al. | Apr 2014 | B2 |
| 8721525 | Heckele et al. | May 2014 | B2 |
| 8750568 | Frank et al. | Jun 2014 | B2 |
| 8780362 | Sharonov et al. | Jul 2014 | B2 |
| 8880151 | Stolka et al. | Nov 2014 | B1 |
| 8892191 | Brennan et al. | Nov 2014 | B2 |
| 8922781 | Tearney et al. | Dec 2014 | B2 |
| 8968347 | McCollam | Mar 2015 | B2 |
| 8988505 | Schaerer et al. | Mar 2015 | B2 |
| 9008757 | Wu | Apr 2015 | B2 |
| 9179984 | Teichman et al. | Nov 2015 | B2 |
| 20010025174 | Daniel et al. | Sep 2001 | A1 |
| 20010027272 | Saito | Oct 2001 | A1 |
| 20010040990 | Dadi | Nov 2001 | A1 |
| 20010055462 | Seibel | Dec 2001 | A1 |
| 20020028986 | Thompson | Mar 2002 | A1 |
| 20020049435 | Mersch | Apr 2002 | A1 |
| 20020049464 | Donofrio et al. | Apr 2002 | A1 |
| 20020087179 | Culp et al. | Jul 2002 | A1 |
| 20020123665 | Miller | Sep 2002 | A1 |
| 20020137987 | Watanabe et al. | Sep 2002 | A1 |
| 20030013960 | Makin et al. | Jan 2003 | A1 |
| 20030029464 | Chen et al. | Feb 2003 | A1 |
| 20030095781 | Williams | May 2003 | A1 |
| 20030118302 | James, IV et al. | Jun 2003 | A1 |
| 20040022527 | Carusillo et al. | Feb 2004 | A1 |
| 20040064019 | Chang et al. | Apr 2004 | A1 |
| 20040122292 | Dey et al. | Jun 2004 | A1 |
| 20040145746 | Kim et al. | Jul 2004 | A1 |
| 20040188616 | Wu et al. | Sep 2004 | A1 |
| 20050004592 | Criscuolo | Jan 2005 | A1 |
| 20050005024 | Samuels et al. | Jan 2005 | A1 |
| 20050054894 | Aizenfeld et al. | Mar 2005 | A1 |
| 20050085718 | Shahidi | Apr 2005 | A1 |
| 20050131426 | Moctezuma de la Barrera et al. | Jun 2005 | A1 |
| 20050135749 | Nield et al. | Jun 2005 | A1 |
| 20060004406 | Wehrstein et al. | Jan 2006 | A1 |
| 20060036162 | Shahidi et al. | Feb 2006 | A1 |
| 20060069314 | Farr | Mar 2006 | A1 |
| 20060114473 | Tearney et al. | Jun 2006 | A1 |
| 20060171693 | Todd et al. | Aug 2006 | A1 |
| 20060235270 | Teichmann | Oct 2006 | A1 |
| 20060235279 | Hawkes et al. | Oct 2006 | A1 |
| 20060241347 | Whitehead | Oct 2006 | A1 |
| 20060258938 | Hoffman et al. | Nov 2006 | A1 |
| 20070060098 | McCoy | Mar 2007 | A1 |
| 20070112336 | Aizenfeld et al. | May 2007 | A1 |
| 20070112337 | Salman et al. | May 2007 | A1 |
| 20070147707 | Coste-Maniere et al. | Jun 2007 | A1 |
| 20070151390 | Blumenkranz et al. | Jul 2007 | A1 |
| 20070167679 | Miyamoto et al. | Jul 2007 | A1 |
| 20070175964 | Shelton, IV et al. | Aug 2007 | A1 |
| 20070179488 | Trusty et al. | Aug 2007 | A1 |
| 20070206275 | Hemmer et al. | Sep 2007 | A1 |
| 20070225550 | Gattani et al. | Sep 2007 | A1 |
| 20070250006 | Court et al. | Oct 2007 | A1 |
| 20070255101 | Bar-Or | Nov 2007 | A1 |
| 20070260121 | Bakos et al. | Nov 2007 | A1 |
| 20080009677 | Shoroji et al. | Jan 2008 | A1 |
| 20080058989 | Oleynikov et al. | Mar 2008 | A1 |
| 20080071140 | Gattani et al. | Mar 2008 | A1 |
| 20080159653 | Dunki-Jacobs et al. | Jul 2008 | A1 |
| 20080167522 | Giordano et al. | Jul 2008 | A1 |
| 20080167671 | Giordano et al. | Jul 2008 | A1 |
| 20080188716 | Heckele et al. | Aug 2008 | A1 |
| 20080208006 | Farr | Aug 2008 | A1 |
| 20080208041 | Gilboa | Aug 2008 | A1 |
| 20080319311 | Hamadeh | Dec 2008 | A1 |
| 20090017430 | Muller-Daniels et al. | Jan 2009 | A1 |
| 20090054767 | Telischak et al. | Feb 2009 | A1 |
| 20090192519 | Omori | Jul 2009 | A1 |
| 20090203991 | Papaioannou et al. | Aug 2009 | A1 |
| 20090209998 | Widmann | Aug 2009 | A1 |
| 20090218527 | French et al. | Sep 2009 | A1 |
| 20090225320 | Bendall | Sep 2009 | A1 |
| 20090240111 | Kessler et al. | Sep 2009 | A1 |
| 20090244260 | Takahashi et al. | Oct 2009 | A1 |
| 20090248041 | Williams et al. | Oct 2009 | A1 |
| 20090270682 | Visser | Oct 2009 | A1 |
| 20090318756 | Fisher et al. | Dec 2009 | A1 |
| 20090318758 | Farr et al. | Dec 2009 | A1 |
| 20090318763 | Koerner et al. | Dec 2009 | A1 |
| 20090323053 | Furman | Dec 2009 | A1 |
| 20100036393 | Unsworth | Feb 2010 | A1 |
| 20100065605 | Shelton, VI et al. | Mar 2010 | A1 |
| 20100065609 | Schwemberger | Mar 2010 | A1 |
| 20100069920 | Naylor et al. | Mar 2010 | A1 |
| 20100096435 | Fuchs et al. | Apr 2010 | A1 |
| 20100106015 | Norris | Apr 2010 | A1 |
| 20100200636 | Zemlok et al. | Aug 2010 | A1 |
| 20100228092 | Ortiz et al. | Sep 2010 | A1 |
| 20100268067 | Razzaque et al. | Oct 2010 | A1 |
| 20110069159 | Soler et al. | Mar 2011 | A1 |
| 20110110114 | Papac et al. | May 2011 | A1 |
| 20110112377 | Papac et al. | May 2011 | A1 |
| 20110161054 | Woolf | Jun 2011 | A1 |
| 20110165535 | Berger et al. | Jul 2011 | A1 |
| 20110208004 | Feingold et al. | Aug 2011 | A1 |
| 20110237915 | Yamaguchi | Sep 2011 | A1 |
| 20110275901 | Shelton, IV | Nov 2011 | A1 |
| 20110282160 | Bhadri et al. | Nov 2011 | A1 |
| 20120024934 | Shelton, IV et al. | Feb 2012 | A1 |
| 20120062724 | Yokota | Mar 2012 | A1 |
| 20120071711 | Shelton, IV et al. | Mar 2012 | A1 |
| 20120071757 | Salcudean et al. | Mar 2012 | A1 |
| 20120082970 | Pravong et al. | Apr 2012 | A1 |
| 20120101497 | Jayaraj | Apr 2012 | A1 |
| 20120108901 | Sargeant et al. | May 2012 | A1 |
| 20120116369 | Viola | May 2012 | A1 |
| 20120130162 | Dolt et al. | May 2012 | A1 |
| 20120143049 | Neubauer et al. | Jun 2012 | A1 |
| 20120184951 | Viola | Jul 2012 | A1 |
| 20120209123 | King | Aug 2012 | A1 |
| 20120238808 | Teichtmann | Sep 2012 | A1 |
| 20120265009 | Yang et al. | Oct 2012 | A1 |
| 20120265010 | Uram | Oct 2012 | A1 |
| 20120265071 | Berke | Oct 2012 | A1 |
| 20120296163 | Stopek | Nov 2012 | A1 |
| 20120302828 | Toledo-Crow et al. | Nov 2012 | A1 |
| 20130038836 | Smith | Feb 2013 | A1 |
| 20130053782 | Shelton, IV | Feb 2013 | A1 |
| 20130053835 | Bacher et al. | Feb 2013 | A1 |
| 20130060084 | Fouts et al. | Mar 2013 | A1 |
| 20130070070 | Katakura | Mar 2013 | A1 |
| 20130103023 | Monson et al. | Apr 2013 | A1 |
| 20130110005 | Sharonov | May 2013 | A1 |
| 20130110006 | Sharonov et al. | May 2013 | A1 |
| 20130110129 | Reid et al. | May 2013 | A1 |
| 20130123800 | Leroy et al. | May 2013 | A1 |
| 20130144267 | Chan et al. | Jun 2013 | A1 |
| 20130226037 | Pinto et al. | Jun 2013 | A1 |
| 20130190759 | Waaler et al. | Jul 2013 | A1 |
| 20130197317 | Daniel et al. | Aug 2013 | A1 |
| 20130218024 | Boctor et al. | Aug 2013 | A1 |
| 20130226156 | Sharonov | Aug 2013 | A1 |
| 20130267787 | Warnock | Oct 2013 | A1 |
| 20130267975 | Timm et al. | Oct 2013 | A1 |
| 20130281845 | Luken | Oct 2013 | A1 |
| 20130296712 | Durvasula | Nov 2013 | A1 |
| 20130317351 | Case et al. | Nov 2013 | A1 |
| 20130317352 | Case et al. | Nov 2013 | A1 |
| 20130317353 | Frank et al. | Nov 2013 | A1 |
| 20130324999 | Price et al. | Dec 2013 | A1 |
| 20130345513 | Tsuruta et al. | Dec 2013 | A1 |
| 20140005484 | Charles | Jan 2014 | A1 |
| 20140005682 | Worrell et al. | Jan 2014 | A1 |
| 20140012078 | Coussa | Jan 2014 | A1 |
| 20140012286 | Lee et al. | Jan 2014 | A1 |
| 20140025067 | Kerr et al. | Jan 2014 | A1 |
| 20140030669 | Hey et al. | Jan 2014 | A1 |
| 20140031665 | Pinto et al. | Jan 2014 | A1 |
| 20140051994 | Graumann et al. | Feb 2014 | A1 |
| 20140052005 | Yokota | Feb 2014 | A1 |
| 20140066784 | Yokota | Mar 2014 | A1 |
| 20140071239 | Yokota | Mar 2014 | A1 |
| 20140071257 | Yokota | Mar 2014 | A1 |
| 20140074116 | Collins | Mar 2014 | A1 |
| 20140088410 | Wu | Mar 2014 | A1 |
| 20140106626 | Frushour et al. | Apr 2014 | A1 |
| 20140107417 | McKinley et al. | Apr 2014 | A1 |
| 20140107471 | Haider | Apr 2014 | A1 |
| 20140107685 | O'Neill et al. | Apr 2014 | A1 |
| 20140114327 | Boudreaux et al. | Apr 2014 | A1 |
| 20140121507 | Nau, Jr. | May 2014 | A1 |
| 20140121508 | Latimer et al. | May 2014 | A1 |
| 20140148799 | Mueller | May 2014 | A1 |
| 20140163359 | Sholev et al. | Jun 2014 | A1 |
| 20140171962 | Kang | Jun 2014 | A1 |
| 20140180001 | Von Grunberg et al. | Jun 2014 | A1 |
| 20140194896 | Frimer et al. | Jul 2014 | A1 |
| 20140200406 | Bennett et al. | Jul 2014 | A1 |
| 20140200561 | Ingmanson et al. | Jul 2014 | A1 |
| 20140204702 | Ratering | Jul 2014 | A1 |
| 20140207002 | Seow | Jul 2014 | A1 |
| 20140235946 | Smith | Aug 2014 | A1 |
| 20140236177 | Verner et al. | Aug 2014 | A1 |
| 20140243658 | Breisacher et al. | Aug 2014 | A1 |
| 20140275764 | Shen et al. | Sep 2014 | A1 |
| 20140275771 | Henley et al. | Sep 2014 | A1 |
| 20140276097 | Sharonov | Sep 2014 | A1 |
| 20140296636 | Hatano | Oct 2014 | A1 |
| 20140303491 | Shekhar et al. | Oct 2014 | A1 |
| 20140357945 | Duckworth | Dec 2014 | A1 |
| 20140367444 | Williams | Dec 2014 | A1 |
| 20140367591 | Mahou et al. | Dec 2014 | A1 |
| 20150005748 | Sniffin et al. | Jan 2015 | A1 |
| 20150018622 | Tesar et al. | Jan 2015 | A1 |
| 20150049907 | Hong et al. | Feb 2015 | A1 |
| 20150065875 | Friebe | Mar 2015 | A1 |
| 20150069108 | Williams | Mar 2015 | A1 |
| 20150073398 | Toledo-Crow et al. | Mar 2015 | A1 |
| 20150076211 | Irka et al. | Mar 2015 | A1 |
| 20150080660 | Gomez et al. | Mar 2015 | A1 |
| 20150080764 | Poe | Mar 2015 | A1 |
| 20150086162 | Miyahara et al. | Mar 2015 | A1 |
| 20150088115 | Smith | Mar 2015 | A1 |
| 20150109427 | Wood et al. | Apr 2015 | A1 |
| 20150119637 | Alvarez et al. | Apr 2015 | A1 |
| 20150133960 | Lohmeier et al. | May 2015 | A1 |
| 20150173591 | Zheng et al. | Jun 2015 | A1 |
| 20150230866 | Tung et al. | Aug 2015 | A1 |
| 20150238276 | Atarot et al. | Aug 2015 | A1 |
| 20150265792 | Goudra et al. | Sep 2015 | A1 |
| 20150359418 | Feussner et al. | Dec 2015 | A1 |
| 20160081712 | Heniford et al. | Mar 2016 | A1 |
| 20160166345 | Kumar et al. | Jun 2016 | A1 |
| 20160278611 | Power | Sep 2016 | A1 |
| 20160360954 | Rohling et al. | Dec 2016 | A1 |
| 20170055819 | Hansen et al. | Mar 2017 | A1 |
| 20170095269 | Reid et al. | Apr 2017 | A1 |
| 20170105802 | Taraschi et al. | Apr 2017 | A1 |
| 20170172382 | Nir | Jun 2017 | A1 |
| 20170238962 | Hansen et al. | Aug 2017 | A1 |
| 20180014851 | Hansen et al. | Jan 2018 | A1 |
| 20180042686 | Peine | Feb 2018 | A1 |
| 20180325604 | Atarot | Nov 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 2 603 353 | Sep 2007 | CA |
| 2533818 | Feb 2003 | CN |
| 2691487 | Apr 2005 | CN |
| 1729938 | Jun 2005 | CN |
| 201316257 | Sep 2009 | CN |
| 201393995 | Feb 2010 | CN |
| 201393999 | Feb 2010 | CN |
| 201602746 | Oct 2010 | CN |
| 101991399 | Mar 2011 | CN |
| 201861616 | Jun 2011 | CN |
| 102401646 | Apr 2012 | CN |
| 102626301 | Aug 2012 | CN |
| 103299355 | Sep 2013 | CN |
| 203379100 | Jan 2014 | CN |
| 101305901 | Apr 2014 | CN |
| 203852327 | Oct 2014 | CN |
| 20002770 | Oct 2000 | DE |
| 102004008488 | Sep 2004 | DE |
| 102005045706 | Apr 2007 | DE |
| 102007007742 | Aug 2008 | DE |
| 102008056830 | May 2010 | DE |
| 202011103007 | Nov 2011 | DE |
| 102012209448 | Dec 2013 | DE |
| 102012209450 | Dec 2013 | DE |
| 2 412 290 | Feb 2012 | EP |
| 1961372 | Mar 2012 | EP |
| 2 554 103 | Feb 2013 | EP |
| 2 630 915 | Aug 2013 | EP |
| 2630915 | Aug 2013 | EP |
| 2551698 | Sep 2014 | EP |
| 3056934 | Aug 2016 | EP |
| 2000-131623 | May 2000 | JP |
| 2001-025469 | Jan 2001 | JP |
| 2003-061970 | Mar 2003 | JP |
| 2006-271600 | Oct 2006 | JP |
| 2015-073663 | Apr 2015 | JP |
| WO 9510218 | Apr 1995 | WO |
| 0042906 | Jul 2000 | WO |
| 0152720 | Jul 2001 | WO |
| WO 02080773 | Oct 2002 | WO |
| 2005122940 | Dec 2005 | WO |
| 2009116969 | Sep 2009 | WO |
| WO 2009116969 | Sep 2009 | WO |
| 2009134634 | Nov 2009 | WO |
| WO 2009134634 | Nov 2009 | WO |
| 2012072112 | Jun 2012 | WO |
| 2012083247 | Jun 2012 | WO |
| WO 2012083247 | Jun 2012 | WO |
| 2013002050 | Jan 2013 | WO |
| 2013096896 | Jun 2013 | WO |
| 2013163391 | Oct 2013 | WO |
| WO 2013163391 | Oct 2013 | WO |
| WO 2014140813 | Sep 2014 | WO |
| 2014174726 | Oct 2014 | WO |
| 2014198675 | Dec 2014 | WO |
| 15124159 | Aug 2015 | WO |
| WO 2015133958 | Sep 2015 | WO |
| WO 2015135055 | Sep 2015 | WO |
| WO 2015149041 | Oct 2015 | WO |
| 2016019424 | Feb 2016 | WO |
| WO 2016018815 | Feb 2016 | WO |
| WO 2016057969 | Apr 2016 | WO |
| Entry |
|---|
| European Patent Application No. 16853132.5, by 3dintegrated ApS: Supplementary European Search Report, dated Oct. 27, 2017 (4 pages). |
| European Patent Application No. 16781653.7, by 3D Integrated ApS: Supplementary European Search Report, dated May 22, 2017 (4 pages). |
| Albitar, C. et al. (2007) “Robust Structured Light Coding for 3D Reconstruction” Proceedings of the 2007 IEEE 11th International Conference on Computer Vision (ICCV 2007), Oct. 14-21, 2007, Rio de Janeiro, Brazil; pp. 1-6, DOI: 10.1109/ICCV.2007.4408982. |
| Edgcumbe, P. et al. (2014) “Pico Lantern: A Pick-up Projector for Augmented Reality in Laparoscopic Surgery” Medical Image Computing and Computer-Assisted Intervention (MICCAI) 17th International Conference, Boston, MA, USA, Sep. 14-18, 2014, Proceedings. P, Golland et al. (Eds.): MICCAI 2014, Part 1, LNCS 8673, pp. 432-439. |
| Hao, W. (Dec. 2011) Wide Baseline Stereo Image Rectification and Matching. Dissertation. The University of Tennessee, Knoxville, Tennessee; 175 pages. |
| Kang, H. (May 2002) Robotic Assisted Suturing in Minimally Invasive Surgery. Thesis. Rensselaer Polytechnic Institute, Troy, New York; 183 pages. |
| Krupa, A. et al. (2002) “Autonomous retrieval and positioning of surgical instruments in robotized laparoscopic surgery using visual servoing and laser pointers” Proceedings of the 2002 IEEE International Conference on Robotics and Automation, May 11-15, 2002, Washington, DC: vol. 4, pp. 3769-3774. |
| Louw, M. and F. Nicolls (2005) “An approximate EM Homographical Iterative Closest Point algorithm” PRASA2005, Langebaan, Cape Town; p. 89-92 [online]. Available from: http://dip.ee.uct.ac.za/˜nicolls/publish/ml05-prasa.pdf. |
| Pellicanò, N. et al. (2016) “Robust Wide Baseline Pose Estimation from Video (Supplementary Material)” Proceedings of the International Conference on Pattern Recognition (ICPR), Dec. 4-8, 2016; 2 pages. |
| Pritchett, P. and A. Zisserman (1998) “Wide Baseline Stereo Matching” Proceedings of the Sixth International Conference on Computer Vision, Jan. 4-7, 1998: pp. 754-760. |
| Reiter, A. et al. (2014) “Surgical Structured Light for 3D Minimally Invasive Surgical Imaging” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014); pp. 1282-1287. |
| Zhang, K. et al. (Sep. 2009) “A Surface Topology and Motion Compensation System for Microsurgery Guidance and Intervention based on Common-Path Optical Coherence Tomography” IEEE Trans Biomed Eng, 56(9):2318-2321. NIH Public Access Author Manuscript; available in PMC Sep. 1, 2010 (12 pages). |
| Nicolau, S. (2011) “Augmented reality in laparoscopic surgical oncology” Surgical Oncology, pp. 189-201. |
| Hyunseok et al. (2015) “An effective visualization technique for depth perception in augmented reality surgical navigation” The International Journal of Medical Robotics and Computer-Assisted Surgery (May 5, 2015) doi: 10.1002/rcs. 1657. |
| Karjee et al, (2014) “Spatial Data Estimation in Three Dimensional Distributed Wireless Sensor Networks” Embedded System (ICES) 2014 International Conference (Jul. 3-4), IEEE ISBN 978-1-4799-5025-6. |
| International Search Report and Written Opinion for PCT/DK2016/050180 dated Apr. 11, 2016 (18 pages). |
| Salvi et al. (2004) “Pattern codification strategies in structured light systems” Pattern Recognition, vol. 37, Issue 4 (Apr.), pp. 827-849. |
| Ackerman, J.D. et al. (2002) “Surface reconstruction of abdominal organs using laparoscopic structured light for augmented reality” Proceedings of SPIE, 4661:39-46. |
| Bauer, S. et al. (2013) “Real-Time Range Imaging in Health Care: A Survey” in Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Dagstuhl Seminar 2012 and GCPR Workshop on Imaging New Modalities. M. Grzegorzek et al. (Eds.) Springer Berlin Heidelberg. Lecture Notes in Computer Science, vol. 8200; pp. 188-212. |
| Clancy, N.T. et al. (2011) “Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging” Biomedical Optics Express, 2(11):3119-3128. |
| European Patent Application No. 17169285.8 by 3dintegrated ApS: Extended European Search Report and European Search Opinion, dated Aug. 31, 2017 (7 pages). |
| Geng, J. (2011) “Structured-light 3D surface imaging: a tutorial” Advances in Optics and Photonics, 3:128-160. |
| Maier-Hein, L. et al. (2013) “Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery” [online]. Author deposited preprint: Retrieved from Sheffield Hallam University Research Archive (SHURA) at: htt..ilshura shu ac.ukI7180i; deposited on Aug. 2, 2013, 58 pages including cover. Final publication in: Medical Image Analysis, 17(8):974-996. (60 Pages). |
| Maier-Hein, L. et al. (2014) “Comparative Validation of Single-shot Optical Techniques for Laparoscopic 3D Surface Reconstruction” Accepted article, IEEE Transactions on Medical Imaging, doi: 10.1109/TMI.2014.2325607, 18 pages. Final publication in vol. 33, Issue 10, pp. 1913-1930, Oct. 2014. |
| Maurice, X. et al. (2013) “Real-time structured light coding for adaptive patterns” J. Real-Time Image Proc, 8:169-178. |
| Schmalz, C. et al. (2012) “An endoscopic 3D scanner based on structured light” Medical Image Analysis, 16:1063-1072. |
| Extended European Search Report received for European Patent Application No. 17884041.9, dated Aug. 12, 2020, 11 pages. |
| Pennington et al. (Jul. 2001) “Miniaturized 3-D Surface Profilometer using a Fiber Optic Coupler”, Optics & Laser Technology, 33(5):313-320. |
| Schick et al. (May 26, 2011) “3D Measuring in the Field of Endoscopy”, Proceedings of SPIE, 8082:808216-1-808216-12. |
| Su et al. (Aug. 26, 2015) “Three-dimensional Shape Measurements using Endoscopes”, Proceedings of SPIE, 9586:95861H-1-95861H-6. |
| Number | Date | Country | |
|---|---|---|---|
| 20170251900 A1 | Sep 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/DK2016/050180 | Jun 2016 | US |
| Child | 15601276 | US |
