The present disclosure relates to a remote operation system.
In current medical settings, surgical robots (manipulators) aimed at reducing the burden on operators during operations have increasingly been used. For example, a minimally invasive computer-assisted teleoperated surgery system allowing an operator to control one or more master input devices to perform surgical procedures on a patient through movements of associated remotely operated tools has been devised.
When using such surgical robots, however, the operator only has visual information to perform an operation. The operation may therefore be affected by the fact that the operator does not feel with his/her thumb and fingers that which otherwise would be felt.
It is an aspect to provide a novel technology for allowing an operator to perceive a state of a surface of an object touched by a device.
According to an aspect of one or more embodiments, there is provided a remote operation system comprising an operation device that remotely operates a device that acts on an object; a microphone that detects sound caused by contact of the object with the device; a tactile sensation device that is provided in the operation device and generates a tactile sensation; and computing circuitry that calculates an operation associated with the tactile sensation, based on the sound
According to another aspect of one or more embodiments, there is provided a remote operation system comprising an operation device that remotely operates a surgical tool; a microphone that detects sound caused by contact between the surgical tool and an object of a surgical operation; an actuator that generates a tactile sensation; and computing circuitry that calculates a remote operation associated with the tactile sensation, based on the sound.
According to yet another aspect of one or more embodiments, there is provided a remote operation system comprising an operation device that remotely operates a forceps; a microphone that detects sound that propagates through the forceps and that is caused by contact between the forceps and an object of a surgical operation; a voice coil motor that is driven based on the sound to generate a tactile sensation; and computing circuitry that calculates a remote operation associated with the tactile sensation, based on the sound.
As discussed above, since an operator only has visual information to perform an operation, the operation may be affected by the fact that the operator does not feel with his/her thumb and fingers that which otherwise would be felt. For example, the operator may press on a patient's organ without noticing that a forceps has hit an organ. There are thus disadvantages in that operators need to go through extensive training before using such surgical robots and perform an operation based on the training. To overcome such disadvantages, there have been demands for development of functions of giving force sensation and tactile sensation to operators.
For example, a manipulation system in which an automatic operation of a slave manipulator that follows a manual operation of a master manipulator is controlled and in which the slave manipulator is operated primarily under force control by a pneumatically-driven system has been proposed to provide a force sensation to the operator.
In such a manipulation system, however, a relatively large reaction force received from an object is required. Thus, a small bump or contact with an object may not be reflected in the force sensation. It is therefore difficult to perceive the hardness or the state of a surface of an object as a force sensation.
It is therefore an aspect of various embodiments to provide a novel technology for allowing an operator to perceive the state of a surface of an object touched by a device.
In some embodiments, a remote operation system may include an operation device that remotely operates a device that acts on an object; a microphone that detects sound caused by contact of the object with the device; a tactile sensation device that is provided in the operation device and gives a tactile sensation to an operator; and a computing unit that calculates an operation associated with the tactile sensation given by the tactile sensation device. The computing unit calculates the operation on the basis of the sound.
According to this configuration, information on an object, which cannot be sufficiently conveyed as force sensation, can be conveyed as tactile sensation to the operator when operating with the operation device. Note that the tactile sensation may be, for example, a sensation that can convey information on the hardness and the state (irregularity, etc.) of a surface of an object when the object is touched. The object may be, for example, an object to be grasped or cut with a device or a part of a human body to be treated with a surgical tool in an operation.
In some embodiments, the tactile sensation device may include an actuator driven in accordance with the sound detected by the microphone. This configuration enables a sensation when the device has touched the object through remote operation using the operation device to be indirectly conveyed in a form of vibration to the operator.
In some embodiments, the actuator may be a voice coil motor capable of vibrating at least within a range of 200 to 400 Hz. This configuration enables an operator to perceive the hardness of the object depending on the frequency of the vibration.
In some embodiments, the remote operation system may further include a surgical tool being the device that is used for a surgical operation. The microphone may be provided near a position at which the object and the surgical tool come in contact with each other. Thus, the sound caused by contact of the object and the forceps can be accurately detected.
In some embodiments, the microphone may be capable of collecting sound at least within a range of 50 to 1000 Hz. Thus, sound at frequencies that can be caused by contact of the object and the device can be collected.
In some embodiments, the surgical tool may be forceps having a shaft. The microphone may be provided on a main unit that detachably holds the forceps, and detect sound propagating through the forceps.
In some embodiments, the remote operation system may include a separator that separates the main unit in a dirty area from the forceps in a clean area. A propagation member that transmits sound more easily than air may be provided between the separator and the forceps.
In some embodiments, the surgical tool may be forceps having a shaft. An adhesive for fixing a microphone to the shaft may be provided. The microphone may be covered with an adhesive so that a sound collector thereof is not exposed. As a result, external sound including ambient sound is hardly collected by the sound collector, and structure-borne sound propagating through the shaft can be preferentially collected.
In some embodiments, urethane rubber may be provided between the microphone and the shaft. This configuration enables collection of structure-borne sound with high accuracy and shutting out external sound.
In some embodiments, various components described above may be combined. In some embodiments, a method, a device, a system, and the like consistent with the remote operation system may be provided.
Various embodiments allow an operator to feel the state of a surface of an object touched by a device.
Various embodiments will now be described with reference to the drawings. Components, members, and processes that are the same as or equivalent to each other illustrated in the drawings are represented by the same reference numerals, and redundant explanation will not be repeated for conciseness. Embodiments are not limited to those described herein, but are examples, and any feature or any combination of features described in the various embodiments is not necessarily essential.
A remote operation system according to some embodiments may be used for operation of a master manipulator of a master-slave surgical assist robot. An example of the surgical assist robot is one configured to operate forceps used in endoscopic surgery. Note that the application of the remote operation system is not limited to surgical assist robots. For example, the remote operation system may be used as a system for remotely operating robots used in logistics factories or manufacturing facilities. In particular, the remote operation system is suitable for remotely performing a process requiring long hours of delicate work by using a robot.
Note that the term remote not only includes a case where the physical distance between an operator and an object being operated is large but also a case where a master device operated by an operator and a slave device being operated is mechanistically separated from each other. In the latter case, the operation is remote operation even if the master device and the slave device are located close to each other.
The computing circuitry 58 calculates the operation on the basis of the sound detected by the microphone 54. The microphone 54 is located near a contact position Q at which the object O and the forceps 50 come into contact with each other. Thus, the sound caused by contact of the object O and the forceps 50 can be accurately detected.
The present inventor has conducted an intensive study on a structure that can remove the airborne sound So, which is ambient sound, as much as possible and selectively collect the structure-borne sound Si that better reflects the attribute (such as the hardness and the surface state) of the object O that is hit. As illustrated in
A recess in which the sensor 54a is located may be advantageously filled with a filling material so that the airborne sound So does not directly reach the sensor 54a. In the embodiment, an adhesive 66 is used as the filling material. In some embodiments, a propagation member 68 for facilitating propagation of the structure-borne sound Si, which has traveled through the shaft 50a, to the sensor 54a is provided between the microphone 54 and the shaft 50a. The propagation member 68 and the microphone 54 are adhered to each other with the adhesive 66, and the propagation member 68 and the shaft 50a are adhered to each other with an adhesive 70.
In an embodiment, the microphone 54 may be a microelectromechanical system (MEMS) microphone (e.g., ICS-40180 manufactured by InvenSense, Inc.) that can be mounted on thin forceps. The frequency band of sound that can be collected by the MEMS microphone is 60 Hz to 20 kHz, and the supply voltage for the MEMS microphone is 1.5 V to 3.6 V. In an embodiment, the adhesive 66 and the adhesive 70 are hot-melt adhesives having thicknesses of about 0.5 mm to 1 mm. In an embodiment, the adhesives may be an epoxy resin. In an embodiment, the propagation member 68 may be urethane rubber with a hardness of A70 and a thickness of about 5±2 mm. However, embodiments are not limited thereto and, in some embodiments, the propagation member 68 may be any material that better transmits sound than air. Examples of other materials include fluororubber. Furthermore, the materials and the forms of the adhesives are not particularly limited as long as the adhesives can make two members stick to each other. In some embodiments, screws or springs may be used instead of the adhesives so that two members are pressed against each other and thus fixed.
In some embodiments, the microphone 54 is located at the base portion of the shaft 50a, which is well away from the distal end of the forceps. This position is insusceptible to driving sound of wires inside the forceps caused by movement of an operating portion at the distal end of the forceps. Thus, the structure-borne sound Si can be accurately detected.
In some embodiments, remote operation system 100 may further include a slave controller 60 that controls a turning movement and a bending movement of individual components of the manipulator included in the slave device 52, and a master controller 62 that operates with the slave controller 60. More specifically, the master component 1 may include the master controller 62 and the slave component 2 may include the slave controller 60. The master controller 62 controls, in addition to the functions of the computing circuitry 58 described above, movements at the slave device 52 on the basis of information on the movements of individual components of the operation device 10. In some embodiments, the slave controller 60 may include signal processing circuitry 64 that converts information on the sound detected by the microphone 54 for subsequent use by the computing circuitry 58. Specifically, the signal processing circuitry 64 performs filtering with a bandpass filter or noise attenuation using a dead band.
A signal processed by the signal processing circuitry 64 is supplied to the computing circuitry 58 in which a specific wavelength that affects the tactile sensation is enhanced, and then supplied to the tactile sensation device 56. The tactile sensation device 56 includes an actuator driven in accordance with the sound S detected by the microphone 54. This configuration enables conveyance of information on an object, which cannot be sufficiently conveyed as force sensation, as tactile sensation to the operator when operating with the operation device. Note that the tactile sensation is, for example, a sensation that can convey information on the hardness and the state (e.g., irregularity, etc.) of a surface of an object O when the object O is touched. The object O is, for example, an object to be grasped or cut with a device or a part of a human body to be treated with a surgical tool in an operation.
In some embodiments, the actuator may be a voice coil motor capable of vibrating at least within a range of 200 to 400 Hz. Vibratory stimulation at a frequency in the range of 200 to 400 Hz is suitable for causing an operator to perceive the hardness of an object O. This configuration enables a sensation when the forceps 50 have touched the object O through remote operation using the operation device 10 to be indirectly conveyed in a form of vibration to the operator. This configuration also enables the operator to perceive the hardness of the object depending on the frequency of the vibration. Note that, in some embodiments, a piezoelectric element may be used as the actuator.
In some embodiments, the microphone 54 is capable of collecting sound at least within a range of 50 to 1000 Hz. This configuration enables collection of sound at a frequency correlated with the hardness, which can be generated when the object O and the forceps 50 come into contact with each other. Suitable examples of the microphone 54 include an MEMS microphone. However, embodiments are not limited thereto. In some embodiments, a piezoelectric vibration sensor or an acceleration sensor may be used as a detector for detecting sound vibration. In some embodiments, a plurality of microphones and sensors may be combined to constitute a detector.
The mechanism of the operation device 10 is not particularly limited and, in some embodiments, the mechanism of the operation device 10 may include joints of multiple degrees of freedom for use as a haptic device. In some embodiments, the mechanism of the slave device 52 may include manipulator arms of multiple degrees of freedom for changing the position and the posture of the forceps 50. In some embodiments, a pneumatic drive system capable of conveying force sensation may also be combined in the mechanism of the operation device 10 or the slave device 52.
This configuration not only enables the whole operation device 10 to be moved with a power grip but also enables the pinch portion 14 to be turned and slid while the grip portion 12 is held by the palm. Thus, functions of both power grip operations and pinch grip operations can be provided, which reduces the burden on the operator's operation and enables delicate work.
The grip portion 12 includes a first hold 12a, having a cylindrical shape, to be directly held with a palm, and a turnable portion 12b that is turnable relative to the first hold 12a. The first hold 12a has a columnar hole in which the turnable portion 12b having a columnar shape is rotatably accommodated. The pinch portion 14 includes a second hold 14a to be directly held with a thumb and fingers, and a guide 14b that guides the second hold 14a to move away from and toward the grip portion 12.
This configuration enables the turnable portion 12b to turn in a state in which the first hold 12a of the grip portion 12 is fixed by the palm. In some embodiments, the second hold 14a can be moved away from and toward the grip portion 12 with the thumb and fingers pinching the second hold 14a.
The turnable portion 12b is a rotating shaft provided in the first hold 12a. The guide 14b is connected to the rotating shaft, which is the turnable portion 12b, and fixed. In some embodiments, the guide 14b includes a linear guide rail 14c extending in a direction intersecting the axis A of the rotating shaft. This configuration enables the second hold 14a to be moved in predetermined directions (in the directions of arrows S1) relative to the grip portion 12. While the angle between the axis A and the extending direction of the guide rail 14c is 90° in the operation device 10 according to the embodiment, the angle may be other than 90° and may be set within a range of 90°+45°, within a range of 90°+30°, or within a range of 90°+15°.
In some embodiments, the second hold 14a is guided by the guide rail 14c so as to move ±5 mm or more with respect to a reference point P relative to the rotating shaft. In other words, the second hold 14a is designed to be movable at least within a range of 10 mm along the guide rail 14c.
The second hold 14a of the operation device 10 is movably supported by a gimbal 20. The gimbal 20 is turnable about three rotation axes that (orthogonally) intersect each other. The gimbal 20 is turnable about an X axis Ax, a Y axis Ay, and a Z axis Az.
Specifically, an end of the second hold 14a in the Z-axis direction is connected to a Z-axis portion 22, and the Z-axis portion 22 is rotatably held by a first arm 24 having an L shape. The first arm 24 also rotatably holds a Y-axis portion 26 and is rotatable relative to a second arm 28 connected with the Y-axis portion 26. The second arm 28 has an L shape, and rotatably holds an X-axis portion 30 at an end opposite the end at which the Y-axis portion 26 is held. The X-axis portion 30 is connected to a translation mechanism such as an XYZ stage or an XYZ link mechanism for movement in three-dimensional directions.
As a result, the movement information of the operation device 10 can be used for control of the position and posture of the manipulator of the slave device via the gimbal 20 and the translation mechanism. For example, in a case where the slave device is a surgical assist robot capable of manipulating forceps, the operator holds the grip portion 12 to move or turn the whole operation device 10 in a desired direction or to a desired orientation. The movement of the operation device 10 is detected by sensors (not illustrated) that obtain information (movement information) on the turning angles of the Z-axis portion 22, the Y-axis portion 26, and the X-axis portion 30. The movement information detected by the sensors is transmitted to the surgical assist robot. The surgical assist robot controls the movement of the position of the forceps on the basis of the transmitted movement information.
Note that the second hold 14a is slidable relative to the grip portion 12 with respect to a point (C in
The remote operation system 100 including the operation device 10 illustrated in
(1) The remote operation system 100 allows an operator to perceive a hit (impact) and a state of a surface of an object;
(2) The remote operation system 100 allows an operator to perceive even a contact and a change thereof causing a force sensation with a magnitude of nearly zero (which would be undistinguishable from errors);
(3) The remote operation system 100 increases the accuracy of an operator's perception of the texture and the hardness of an object being operated by combining a force sensation conveying mechanism in addition to a tactile sensation conveying mechanism; and
(4) The remote operation system 100 provides event information (an alarm or operation guidance) of a system in a form of tactile sensation without interfering with an operation or eyesight.
A case in which the microphone is attached to the shaft with another member therebetween has been described. However, in some embodiments, a configuration in which a microphone is provided on a main unit of a slave device to/from which forceps are attachable/detachable may be used.
Various embodiments as described above may be applied to a remote operation system for assembly or sorting of products, surgery, or the like.
While various embodiments have been described above with reference to the drawings, embodiments are not limited to those described above, and any combination or substitution of components is included in the scope of the appended claims. In addition, modifications such as combinations, changes in the order of processes, and various changes in design in the embodiments may be made on the embodiments on the basis of knowledge of a person skilled in the art, and such modified embodiments are included within the scope of the appended claims.
This application is a continuation application of International Patent Application No. PCT/JP2022/019244 filed on Apr. 28, 2022 in the Japan Patent Office, the contents of which being incorporated by reference herein in its entirety.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2022/019244 | Apr 2022 | WO |
Child | 18925189 | US |