The present invention is directed towards systems and methods for providing haptic feedback to the operator of a surgical system, and more particularly to maintaining an intuitive haptic profile for the user when haptic feedback actuators reach performance limits.
Tele-operated surgical systems are often intended to enhance surgeon precision and/or reduce patient trauma during a medical procedure. In such systems, a surgeon interacts with input devices (sometimes referred to as “masters” or “master controllers”) to control surgical instruments that are actuated by drive mechanisms such as motors. Because the surgeon is not directly manipulating the surgical instruments, in can sometimes be beneficial to provide haptic feedback at the input devices that indicates or replicates interaction forces (e.g., felt at the surgical instruments, other elements of the surgical system, and/or virtual or synthetic elements/features generated by the surgical system). Note that the force feedback being presented to the user may be the sum of the feedback from a sensor, from an algorithm, from a user interface cue, collision detection, model interaction, etc.
To provide a good user experience, the surgeon would ideally experience a seamless haptic experience through system state and configuration changes. However, this can be difficult to accomplish, for example the actuators used to provide the haptic feedback are have different performance limits or if different actuators reach their performance limits at different times. In such instances, the haptic feedback presented to the user may not properly align with the perceived user experience (e.g., actual force being sensed at the surgical instrument or the visual representation of the interactive object), resulting in a confusing or non-intuitive user experience.
For example, motor torque limits (i.e. saturation limits) are typically imposed in software for robotic interfaces. These performance limits can be created for several reasons including to protect the motor from overheating, to limit the force applied to the surgeon/patient, and/or to keep the motor in it's ideal torque operating range. Limiting the torque at the motor can yield non-isotropic force saturation at the interface (e.g., the handle of the input device or the tip of the instrument). This means when you are trying to present a force at the interface that involves two or more motors, the force direction can be incorrect if one motor is restricted by a motor torque limit.
This can be especially problematic when rendering force to the user (haptic feedback) at the input device. The user may be feeling a force in a given direction, but as the force applied to the user increases and reaches a torque limit for any of the motors associated with render the force to the user, the direction of the force displayed to the user begins to rotate which can be disconcerting and/or confusing for the user.
It is therefore desirable to provide a system and method for ensuring haptic feedback that is consistent with the force environment at the surgical instrument.
To minimize discrepancies between expected force directions and haptic force feedback directions, the outputs of the haptic feedback actuators are scaled whenever one or more of the actuators reaches a predetermined output threshold, thereby maintaining proper haptic feedback directionality when individual actuators would otherwise be commanded to operate outside their accurate performance range. Such scaling may change the overall haptic feedback magnitude, but allows the haptic feedback direction to be appropriately maintained.
In some embodiments, where software limits the output of any actuator, a monitoring process can determine when an output of an actuator would exceed a maximum threshold output (at or below the software-defined output limit), and at that point scale down the output of at least one of the other actuators so that the desired direction of the total output (e.g., force or torque) is maintained. In some embodiments, the monitoring process can additionally or alternatively determine when an output of an actuator would fall below a minimum output threshold (below which the actuator output may be too low to generate accurately), and at that point scale up its output and that of at least one of the other actuators so that the desired direction of the total output (e.g., force or torque) is maintained. In some embodiments, all actuators are scaled when one reaches its threshold output, while in other embodiments, only concurrently active actuators are scaled. In some embodiments, the output thresholds for the actuators are fixed, and in various other embodiments, the output thresholds for the actuators can vary over time or based on actuator and/or system status.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.
Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is emphasized that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.
In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one skilled in the art that the embodiments of this disclosure may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention. And, to avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments.
To minimize discrepancies between desired (e.g., sensed or modeled by a surgical system) and actual haptic force feedback directions, the outputs of the haptic feedback actuators are scaled whenever a commanded output of one or more of the actuators falls outside a predetermined working range for that actuator. The predetermined working range can be defined by a maximum output threshold equal to or less than the output limit of the actuator, and/or a minimum output threshold equal to or greater than a minimum accurate output level of the actuator. Such scaling may reduce the overall haptic feedback magnitude, but allows the haptic feedback direction to be appropriately maintained, which is often a more critical aspect of haptic feedback.
The actual force feedback provided at the input device is generated by two or more actuators (e.g., motors, drives, or any other motive elements) that work cooperatively to provide feedback of varying force and direction. For example, an input device having pitch and yaw capabilities may be coupled to a first pair of actuators that apply forces in opposing directions about the pitch axis, and a second pair of actuators that apply forces in opposing directions about the yaw axis. Two or more of the pitch and yaw actuators can then be used simultaneously to provide force feedback that is offset from the pitch and yaw axes.
Because actuators generally do not have performance characteristics that are exactly the same, whether due to inherent performance limitations or due to operational constraints/effects (e.g., heat, mechanical restrictions). For example, one out of the group of haptic feedback actuators will typically reach its maximum output level before the others. Any commanded output beyond that maximum output level will not result in any increased output, and consequently any haptic feedback that involves the maxed-out actuator will likely deviate from the expected force feedback direction. Additionally, the actuators will may begin producing noisier (less precise) outputs as outputs decrease below a certain level, such that haptic feedback below a certain level may also deviate from the expected force feedback direction.
As noted above, the actuators cannot exceed their output limits, which may be static (e.g., defined by inherent performance characteristics of the actuators), or dynamic (e.g., based on a current parameter of an actuator, such as temperature, or a physical restriction such as a kinematic configuration of the driven structure that places the actuator in a mechanically unfavorable position).
The problem thus arises if a commanded output of an actuator exceeds its output limit, such as indicated by desired (commanded) feedback force FD. To generate desired feedback force FD, actuator 1 receives a commanded output CO1, and actuator 2 receives a commanded output CO2. Commanded output CO1 is less than output limit OL1, and so can be provided by actuator 1. However, because commanded output CO2 is greater than output limit OL2, the actual output of actuator 2 will be limited to output level OL2, resulting in an overall feedback force FO that is both smaller than, and offset from, desired feedback force FD. While a slight variability in the magnitude of haptic feedback can generally be accommodated by a user without much difficulty, a deviation in force direction can be quite misleading to a user and can result in improper control actions by the user in response.
Returning to
Then, in a GLOBAL ACTUATOR SCALING step 130, a common scaling factor is applied to the commanded outputs for the actuators. The scaling factor is selected to keep the output of the identified actuator (i.e., the actuator identified in step 120 having a commanded output greater than its output threshold) less than its output limit. Because the scaling factor is applied to the commanded output for each actuator, the direction of the overall force is maintained, with a reduced overall magnitude. Note that in some embodiments, if multiple actuators would receive commanded outputs that would exceed their output limits, the scaling factor would be based on the commanded output that exceeds its associated actuator's output threshold by the greatest amount—i.e., the scaling factor would be based on the “worst” output discrepancy.
In some embodiments, actuators 1 and 2 could additionally or alternatively exhibit reduced output accuracy at low output levels. In such embodiments, a scaling factor could be applied to commanded outputs CO1 and CO2 if either is less than a minimum output threshold for actuator 1 or actuator 2, respectively. The scaling factor would then increase the adjusted commanded outputs CO1′ and CO2′ above the level at which output accuracy is degraded. Note that as described above, the minimum output thresholds for the haptic feedback actuators can be either static or dynamic, and can be individualized or common across the actuators.
In any event, the application of the scaling factor results in an overall scaled feedback force FS that is aligned directionally with the original desired feedback force FD. As noted above, a consistent haptic experience can be provided even with the change in force magnitude so long as directional consistency of the force feedback is maintained.
Returning to
As noted previously, in some embodiments, the output thresholds for the haptic feedback actuators can be dynamic—i.e., the specific values can change depending on actuator operational parameters, input device kinematic configuration, or various other factors. In such embodiments, over the course of operation of the surgical system, the output threshold(s) applied in step 120 can have different values. In addition, in various other embodiments, different actuators can trigger step 120 over the course of operation of the surgical system.
For exemplary purposes,
Although the haptic feedback profile force FD is described as being derived from a force FM sensed at end effector 311 of instrument 310 for exemplary purposes, in various other embodiments, force FM could be sensed at any location for which corresponding haptic feedback at input device 330 would be beneficial, such as interactions at shaft 312 or any other element of manipulation structure 313 (e.g., arm collisions with structures or staff).
In various other embodiments, force FM can be defined according to non-physical parameters, such as guidance or user interface features. For example, in some embodiments, surgical system 300 can include a display 350 (e.g., a monitor(s), a head-in viewer(s), projections, video glasses/helmet(s), and/or any other graphical presentation element). In various embodiments, display 350 can present a virtual or synthetic element 361 that can be interacted with via input device 330. In some embodiments, synthetic element 361 can be used as a supplemental interface for interacting with a physical component of surgical system 300. For example, as shown in
In various other embodiments, surgical system 300 may provide guidance to the user with respect to movement of instrument 310 and/or input device 330. For example, a desired motion of instrument 310 (e.g., a targeted or safe dissection path, a desired retraction movement, or any other beneficial articulation) could optionally be defined as a trajectory 362. By generating a haptic feedback profile based on model forces FM2 associated with maintaining the position of instrument 310 along trajectory 362 (e.g., inwardly directed forces produced upon deviations from trajectory 362), controller 320 can then attempt to provide an appropriate haptic feedback profile force FD at input device 330.
The various actuators 341 provide actuation outputs (e.g., torque or force) in combination with one another in an effort to produce the desired haptic feedback profile force FD. However, if desired haptic feedback profile force FD requires an output from one of actuators 341 that falls outside its working range, attempting to use the commanded outputs for actuators 341 without modification would result in an unmodified haptic feedback force FO having both a magnitude and direction different than desired force FD (as described above with respect to
Therefore, when controller 320 detects that a commanded output for any of actuators 341 would go beyond that actuator's defined output threshold (as described above with respect to step 120 in
In various embodiments, when controller 320 detects that the unscaled commanded outputs would no longer go beyond the defined output thresholds for any of actuators 341, the scaling factor can be eliminated (or set to 1), thereby allowing restoration of unscaled haptic feedback at input device 330 (as described above with respect to steps 140 and 150 in
While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.
This application is a continuation of U.S. patent application Ser. No. 16/069,810, now U.S. Pat. No. 11,376,087, filed Jul. 12, 2018, and published as US 2019/0015169 A1 on Jan. 17, 2019, which is a U.S. National Stage Filing under 35 U.S.C. 371 from International Application No. PCT/US2017/013239, filed Jan. 12, 2017, and published as WO 2017/123797 A1 on Jul. 20, 2017, which claims the benefit of priority to U.S. Provisional Patent Application No. 62/277,827, filed on Jan. 12, 2016, which is hereby incorporated by reference herein in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
4160508 | Frosch et al. | Jul 1979 | A |
4604016 | Joyce | Aug 1986 | A |
5116180 | Fung et al. | May 1992 | A |
5382885 | Salcudean et al. | Jan 1995 | A |
5696837 | Green | Dec 1997 | A |
5710870 | Ohm et al. | Jan 1998 | A |
5767648 | Morel et al. | Jun 1998 | A |
6024576 | Bevirt et al. | Feb 2000 | A |
6102850 | Wang et al. | Aug 2000 | A |
6377011 | Ben-Ur | Apr 2002 | B1 |
6459926 | Nowlin et al. | Oct 2002 | B1 |
6594552 | Nowlin et al. | Jul 2003 | B1 |
6639581 | Moore et al. | Oct 2003 | B1 |
6723106 | Charles et al. | Apr 2004 | B1 |
6994703 | Wang et al. | Feb 2006 | B2 |
7155315 | Niemeyer et al. | Dec 2006 | B2 |
7248944 | Green | Jul 2007 | B2 |
7373219 | Nowlin et al. | May 2008 | B2 |
7843158 | Prisco | Nov 2010 | B2 |
7963288 | Rosenberg et al. | Jun 2011 | B2 |
8041459 | Sutherland et al. | Oct 2011 | B2 |
8170717 | Sutherland et al. | May 2012 | B2 |
8343171 | Farritor et al. | Jan 2013 | B2 |
8390218 | Houston et al. | Mar 2013 | B2 |
8396598 | Sutherland et al. | Mar 2013 | B2 |
8644988 | Prisco et al. | Feb 2014 | B2 |
9002518 | Manzo et al. | Apr 2015 | B2 |
9014856 | Manzo et al. | Apr 2015 | B2 |
9050120 | Swarup et al. | Jun 2015 | B2 |
9173713 | Hart et al. | Nov 2015 | B2 |
9198714 | Worrell et al. | Dec 2015 | B2 |
9220567 | Sutherland et al. | Dec 2015 | B2 |
9339343 | Swarup et al. | May 2016 | B2 |
9649174 | Swarup et al. | May 2017 | B2 |
11357587 | Verner | Jun 2022 | B2 |
11376087 | Verner | Jul 2022 | B2 |
20010018591 | Brock et al. | Aug 2001 | A1 |
20020120188 | Brock et al. | Aug 2002 | A1 |
20030004610 | Niemeyer et al. | Jan 2003 | A1 |
20030060927 | Gerbi et al. | Mar 2003 | A1 |
20040024311 | Quaid, III | Feb 2004 | A1 |
20040039485 | Niemeyer et al. | Feb 2004 | A1 |
20040106916 | Quaid | Jun 2004 | A1 |
20040243147 | Lipow | Dec 2004 | A1 |
20050024331 | Berkley et al. | Feb 2005 | A1 |
20060149418 | Anvari | Jul 2006 | A1 |
20060207978 | Rizun et al. | Sep 2006 | A1 |
20070142968 | Prisco et al. | Jun 2007 | A1 |
20070239028 | Houser et al. | Oct 2007 | A1 |
20090062813 | Prisco et al. | Mar 2009 | A1 |
20090248037 | Prisco | Oct 2009 | A1 |
20090248038 | Blumenkranz et al. | Oct 2009 | A1 |
20100041991 | Roundhill | Feb 2010 | A1 |
20100073150 | Olson | Mar 2010 | A1 |
20100256558 | Olson et al. | Oct 2010 | A1 |
20100274087 | Diolaiti et al. | Oct 2010 | A1 |
20100275718 | Stuart | Nov 2010 | A1 |
20110040306 | Prisco | Feb 2011 | A1 |
20110118748 | Itkowitz | May 2011 | A1 |
20110295247 | Schlesinger et al. | Dec 2011 | A1 |
20120078080 | Foley et al. | Mar 2012 | A1 |
20120116391 | Houser et al. | May 2012 | A1 |
20120184955 | Pivotto et al. | Jul 2012 | A1 |
20120265051 | Fischer et al. | Oct 2012 | A1 |
20130103050 | Richmond et al. | Apr 2013 | A1 |
20130131867 | Olds et al. | May 2013 | A1 |
20130172906 | Olson et al. | Jul 2013 | A1 |
20130211418 | Lim et al. | Aug 2013 | A1 |
20130211590 | Diolaiti et al. | Aug 2013 | A1 |
20130289767 | Lim | Oct 2013 | A1 |
20130296737 | McMillan et al. | Nov 2013 | A1 |
20130304258 | Taylor et al. | Nov 2013 | A1 |
20140081460 | Ando et al. | Mar 2014 | A1 |
20140118126 | Garg | May 2014 | A1 |
20140142592 | Moon et al. | May 2014 | A1 |
20140330073 | Ko et al. | Nov 2014 | A1 |
20150045812 | Seo | Feb 2015 | A1 |
20150130599 | Berkley et al. | May 2015 | A1 |
20150245876 | Kim et al. | Sep 2015 | A1 |
20150360365 | Fudaba | Dec 2015 | A1 |
20190015169 | Verner et al. | Jan 2019 | A1 |
20190201137 | Shelton, IV et al. | Jul 2019 | A1 |
20210093409 | Overmyer et al. | Apr 2021 | A1 |
20230285100 | Kadokura | Sep 2023 | A1 |
20230320802 | Verner et al. | Oct 2023 | A1 |
Number | Date | Country |
---|---|---|
101443162 | May 2009 | CN |
102612350 | Jul 2012 | CN |
103203755 | Jul 2013 | CN |
203092570 | Jul 2013 | CN |
2008541797 | Nov 2008 | JP |
2012513845 | Jun 2012 | JP |
20140065895 | May 2014 | KR |
20150007020 | Jan 2015 | KR |
WO-2006120666 | Nov 2006 | WO |
WO-2013059643 | Apr 2013 | WO |
WO-2013164622 | Nov 2013 | WO |
WO-2017146890 | Aug 2017 | WO |
WO-2023076468 | May 2023 | WO |
Entry |
---|
Office Action for JP Application No. 2018-530684, mailed Jun. 14, 2022, 6 pages. |
Okamura A.M., “Haptic Feedback in Robot-Assisted Minimally Invasive Surgery,” Current Opinion in Urology, Jun. 24, 2009, National Institute of Health Public Access, vol. 19 (1), 10 pages. |
Extended European Search Report for Application No. EP17738964.0 mailed on Aug. 16, 2019, 9 pages. |
Extended European Search Report for Application No. EP21153588.5 mailed on Jul. 9, 2021, 7 pages. |
International Search Report and Written Opinion for Application No. PCT/US2017/013239, mailed on May 1, 2017, 9 pages. |
Vertut, J, and Coiffet, P., “Robot Technology: Teleoperation and Robotics Evolution and Development,” English translation, Prentice-Hall, Inc., Inglewood Cliffs, NJ, USA 1986, vol. 3A, 332 pages. |
Office Action for JP Application No. 2021-188526, mailed Feb. 21, 2023, 20 pages. |
Office Action for Application No. KR1020187019823, mailed Aug. 24, 2023, 5 Pages. |
Number | Date | Country | |
---|---|---|---|
20220273384 A1 | Sep 2022 | US |
Number | Date | Country | |
---|---|---|---|
62277827 | Jan 2016 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16069810 | US | |
Child | 17749799 | US |