Patent Application
20250031968
Tactile sensing can provide increased information for a variety of applications ranging from medical applications such as tumor classification, inspections of industrial and municipal systems such as pipe inspections, and farming applications such as fruit harvesting, for example.
Colorectal and gastric cancers are leading types of cancer worldwide and are the third leading cause of death related to cancer in the US (see A. Bhandari et al., “Colorectal cancer is a leading cause of cancer incidence and mortality among adults younger than 50 years in the USA: a SEER-based analysis with comparison to other young-onset cancers,” vol. 65, no. 2, p. 311). Morphological characteristics (i.e., shape and texture) of gastric and colorectal cancers are well-known to be associated with tumor stage, pathological subtypes, presence of signet-ring cell morphology, incidence of lymph node involvement, and treatment and survival outcomes. A strong association has been reported between tumor morphology and pathologic and molecular characteristics of the tumor such as tumor grade, signet-ringed cell, and microsatellite instability histology. However, evaluation of the shape and texture of the tumor is subjective and accurate classification requires extensive experience and has not achieved international consensus. Therefore, despite its significant potential, tumor geometry and texture has not yet fully been integrated to guide treatment strategy, such as neoadjuvant therapy regimen (e.g., microsatellite instability-high colorectal cancer frequently responds to immune-checkpoint inhibitor, diffuse-type signet ring cell gastric cancer are less likely to respond to conventional chemotherapy).
Furthermore, current robotic surgical platforms still lack providing reliable tactile feedback to a surgeon, which limits the safety and efficacy of the operations. Hence, surgeons rely solely on visual feedback to identify tumor type and location to guide the extent of the resection of colorectal and gastric cancer. Of note, this may increase the risk of missed lesions and/or unnecessarily extended resection of those organs due to the lack of accurate tumor localization.
Haptics-enabled robots that can provide accurate feedback to surgeons are critically needed to improve surgical accuracy and treatment outcomes of colorectal and gastric cancer surgery. Similarly, in endoscopy (colonoscopy and gastroscopy), even with the recent advancement of optics, the technology still lacks tactile sensation for further detection and evaluation of the lesions. Endoscopic evaluation is still technically demanding (in terms of safe introduction and steering of the scope to evaluate the entire colon) and lesions can be easily missed at flexures of the colon or behind folds since the technique is completely dependent solely on visual information.
Tactile Sensing (TS) is the process of perceiving the physical properties of an object through a cutaneous touch-based interaction (see R. S. Dahiya et al., “Tactile sensing for robotic applications,” Sensors, Focus on Tactile, Force and Stress Sensors, pp. 298-304, 2008). The acquired information can be very beneficial in many areas of robotics in which a robot interacts with hard or deformable objects. Examples include object manipulation (see A. Yamaguchi and C. G. Atkeson, “Recent progress in tactile sensing and sensors for robotic manipulation: can we turn tactile sensing into vision?” Advanced Robotics, vol. 33, no. 14, pp. 661-673, 2019), object texture or stiffness recognition (see G. Li et al., “Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition,” Science Robotics, vol. 5, no. 49, p. eabc8134, 2020), human-robot interaction (see J. M. Gandarias et al., “Enhancing perception with tactile object recognition in adaptive grippers for human-robot interaction,” Sensors, vol. 18, no. 3, p. 692, 2018), and robot-assisted minimally invasive surgery (see F. Ju et al., A miniature piezoelectric spiral tactile sensor for tissue hardness palpation with catheter robot in minimally invasive surgery,” Smart Materials and Structures, vol. 28, no. 2, p. 025033, 2019). As the application of TS in robotics is increasing, there is an imminent need for the development of high-resolution and high-accuracy TS devices that can safely and robustly interact with a soft or rigid environment. To address this crucial need, various tactile sensing technologies have been developed by many researchers. Examples include but are not limited to various electrical-based (e.g., piezoresistive, piezoelectric, inductive, and capacitive) and optical-based (e.g., intensity modulation, wavelength modulation, and phase modulation) hardware (see Y. Liu et al., “Recent progress in tactile sensors and their applications in intelligent systems,” Science Bulletin, vol. 65, no. 1, pp. 70-88, 2020) (see M. Park et al., “Recent advances in tactile sensing technology,” Micromachines, vol. 9, no. 7, p. 321, 2018). However, these approaches still mainly suffer from the large size (for optical tactile sensors), poor resolution (for piezoelectric tactile sensors), poor reliability (for inductive tactile sensors), and rigidity of the sensorized instrument making their usage impractical in a sensitive environment such as medical applications (see N. Kattavenos et al., “Force-sensitive tactile sensor for minimal access surgery,” Minimally Invasive Therapy & Allied Technologies, vol. 13, no. 1, pp. 42-46, 2004) (see J. Dargahi et al., “Development and three-dimensional modelling of a biological-tissue grasper tool equipped with a tactile sensor,” Canadian Journal of Electrical and Computer Engineering, vol. 30, no. 4, pp. 225-230, 2005) (see M. S. Arian et al., “Using the biotac as a tumor localization tool,” in 2014 IEEE Haptics Symposium (HAPTICS). IEEE, 2014, pp. 443-448) (see P. S. Wellman et al., “Tactile imaging of breast masses: first clinical report,” Archives of surgery, vol. 136, no. 2, pp. 204-208, 2001) (see N. Wettels et al., “Biomimetic tactile sensor array,” Advanced Robotics, vol. 22, no. 8, pp. 829-849, 2008). Additionally, high interaction forces for providing adequate tactile feedback hinders the use of these TS devices when interacting with deformable and delicate environments (see E. Heijnsdijk et al., “Inter- and intraindividual variabilities of perforation forces of human and pig bowel tissue,” Surgical Endoscopy and other interventional Techniques, vol. 17, no. 12, pp. 1923-1926, 2003) (see W. Othman and M. A. Qasaimeh, “Tactile sensing for minimally invasive surgery: Conventional methods and potential emerging tactile technologies,” Frontiers in Robotics and AI, p. 376, 2021). Discrete or localized low-quality TS measurement area is another drawback of the aforementioned technologies (see F. Bianchi et al, “Endoscopic tactile instrument for remote tissue palpation in colonoscopic procedures,” in 2017 IEEE International Conference on Cyborg and Bionic Systems (CBS). IEEE, 2017, pp. 248-252) (see C.-H. Won et al., “Tactile sensing systems for tumor characterization: A review,” IEEE Sensors Journal, vol. 21, no. 11, pp. 12578-12588, 2021) (see C.-H. Chuang et al., “Piezoelectric tactile sensor for submucosal tumor detection in endoscopy,” Sensors and Actuators A: Physical, vol. 244, pp. 299-309, 2016) (see P. Baki et al., “Design and characterization of' a novel, robust, tri-axial force sensor,” Sensors and actuators A: physical, vol. 192, pp. 101-110, 2013) (see R. Ahmadi et al., “Microoptical force distribution sensing suitable for lump/artery detection,” Biomedical microdevices, vol. 17, no. 1, pp. 1-12, 2015).
Besides the aforementioned TS devices, Vision-based Tactile Sensors (VTSs) have also recently been developed to enhance tactile perception via high-resolution visual information (see A. Yamaguchi and C. G. Atkeson, “Tactile behaviors with the visionbased tactile sensor fingervision,” International Journal of Humanoid Robotics, vol. 16, no. 03, p. 1940002, 2019). Particularly, VTSs can provide qualitative 3D visual image reconstruction and localization of the interacting rigid or deformable objects by capturing very small deformations of an elastic gel layer that directly interacts with the objects' surface (see U. H. Shah et al., “On the design and development of vision-based tactile sensors,” Journal of Intelligent & Robotic Systems, vol. 102, no. 4, pp. 1-27, 2021). The evolution of digital and small size cameras has improved the fabrication, sensitivity, and integration of the VTSs with different robotic systems (see A. C. Abad and A. Ranasinghe, “Visuotactile sensors with emphasis on gelsight sensor: A review,” IEEE Sensors Journal, vol. 20, no. 14, pp. 7628-7638, 2020). Further, advancements in computer vision and machine learning algorithms have enabled real-time post processing and analysis of the high-resolution TS information provided by this sensor (see M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 1070-1077) (see K. Shimonomura, “Tactile image sensors employing camera: A review,” Sensors, vol. 19, no. 18, p. 3933, 2019). GelSight is the most well-known VTS and has been used in various applications such as the surface texture recognition (see R. Li and E. H. Adelson, “Sensing and recognizing surface textures using a gelsight sensor,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1241-1247), geometry measurement with deep learning algorithms (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017), localization and manipulation of small objects (see R. Li et al., “Localization and manipulation of small parts using gelsight tactile sensing,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014, pp. 3988-3993), and hardness estimation (see W. Yuan et al., “Shape-independent hardness estimation using deep learning and a gelsight tactile sensor,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 951-958). Despite these advancements and as one of the major drawbacks of these sensors, most of the current VTSs still can solely provide qualitative visual deformation images and cannot yet directly provide a quantitative measure of the gel layer deformation (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) (see W. K. Do and M. Kennedy, “Densetact: Optical tactile sensor for dense shape reconstruction,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 6188-6194).
To address this limitation, various techniques have been proposed in the literature to provide a quantitative measure of the deformation when a VTS interacts with an object. These approaches can be mainly divided into two classes: the marker-tracking-based and reflective-membrane-based [21], sensors (see U. H. Shah et al., “On the design and development of vision-based tactile sensors,” Journal of Intelligent & Robotic Systems, vol. 102, no. 4, pp. 1-27, 2021) (see W. Kim et al., “Uvtac: Switchable uv marker-based tactile sensing finger for effective force estimation and object localization,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6036-6043, 2022). For instance, GelForce (see K. Kamiyama et al., “Vision-based sensor for real-time measuring of surface traction fields,” IEEE Computer Graphics and Applications, vol. 25, no. 1, pp. 68-75, 2005) (see K. Sato et al., “Finger-shaped gelforce: sensor for measuring surface traction fields for robotic hand,” IEEE Transactions on Haptics, vol. 3, no. 1, pp. 37-47, 2009), and Chromatouch (see X. Lin et al., “Curvature sensing with a spherical tactile sensor using the color-interference of a marker array,” in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 603-609) (see X. Lin and M. Wiertlewski, “Sensing the frictional state of a robotic skin via subtractive color mixing,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2386-2392, 2019), are examples of marker-tracking-based sensors, in which a pattern of markers is utilized within the elastomer body. When the interaction occurs between the sensing gel layer and the object, the markers' pattern is affected, and their movements can be processed to infer the tactile information. However, marker-based designs require an arduous and complex manufacturing procedure to robustly adhere and integrate the markers with the VTS gel layer. Examples of these procedures include casting or 3D-printing of gel layers (see U. H. Shah et al., “On the design and development of vision-based tactile sensors,” Journal of Intelligent & Robotic Systems, vol. 102, no. 4, pp. 1-27, 2021) as well as relatively easier and inaccurate marker-printing approaches based on transfer papers or handwriting (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) (see W. Yuan et al., “Shape-independent hardness estimation using deep learning and a gelsight tactile sensor,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 951-958). Nevertheless, these techniques suffer from wrinkling or inconsistencies in the printing pattern, which directly deteriorates the accuracy of the deformation measurements. On the contrary, reflective membrane-based sensors, called retrographic sensors, are employed for sensing the shape and texture of the objects through the analysis of the intensity change of the reflected light from the reflective elastic sensing surfaces (see U. H. Shah et al., “On the design and development of vision-based tactile sensors,” Journal of Intelligent & Robotic Systems, vol. 102, no. 4, pp. 1-27, 2021). GelSight is the typical example of the reflective membrane-based VTS that can sense the deformations occurred on the sensor gel layer by the reconstruction of a high-resolution heightmap using colored LED lights, and photometric stereo (see M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 1070-1077). Despite the benefits of using this technique, the flat surface of the gel layer can limit the sensing of the object's orientation. Moreover, tactile information regarding tangential forces and deformation cannot be accurately detected in this approach, and objects without textures or edges are not recognized very well in reflective membrane-based sensors (see W. Kim et al., “Uvtac: Switchable uv marker-based tactile sensing finger for effective force estimation and object localization,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6036-6043, 2022). To address the challenges associated with these two types of VTSs, Nozu et al. (see K. Nozu and K. Shimonomura, “Robotic bolt insertion and tightening based on in-hand object localization and force sensing,” in 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM). IEEE, 2018, pp. 310-315) developed a tactile sensor incorporating both reflective membrane-based sensing and marker-based design for in-hand object localization and force sensing. However, in this approach, utilized markers on the surface of the gel layer may occlude the view of the camera looking towards the reflective membrane, and therefore, deteriorating the quality of the extracted tactile information from the visual output. To mitigate this issue, recently, Kim et al. (see W. Kim et al., “Uvtac: Switchable uv marker-based tactile sensing finger for effective force estimation and object localization,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6036-6043, 2022) proposed a tactile sensor that merges both reflective membrane-based sensing and marker-based design to decouple the marker and reflective membrane images and offer a 3-axis force estimation and object localization. Similar to the previous VTS, however, this sensor only provides an estimated interaction force between the VTS and the object and does not provide quantitative information on the deformation of the gel layer.
Amid the tactile sensors, Vision-based Tactile Sensors (VTSs) have recently been developed to improve tactile perception via high-resolution visual information (see J. Zhu et al., “Challenges and outlook in robotic manipulation of deformable objects,” arXiv preprint arXiv: 2105.01767, 2021). VTSs can provide high-resolution 3D visual image reconstruction and localization of the interacting objects by capturing tiny deformations of an elastic gel layer that directly interacts with the objects' surface (see U. H. Shah at al. “On the design and development of vision-based tactile sensors,” Journal of Intelligent & Robotic Systems, vol. 102, no. 4, pp. 1-27, 2021). GelSight is the most well known VTS, developed by Johnson and Adelson (see M. K. Johnson and E. H. Adelson, “Retrographic sensing for the measurement of surface texture and shape,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009, pp. 1070-1077), and has been utilized for various applications, including the surface texture recognition (see R. Li and E. H. Adelson, “Sensing and recognizing surface textures using a gelsight sensor,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1241-1247), geometry measurement with deep learning algorithms (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017), localization and manipulation of small objects (see R. Li et al. “Localization and manipulation of small parts using gelsight tactile sensing,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014, pp. 3988-3993, and hardness estimation (see W. Yuan et al., “Shape-independent hardness estimation using deep learning and a gelsight tactile sensor,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 951-958). More details about the use of VTSs can be found in A. C. Abad and A. Ranasinghe, “Visuotactile sensors with emphasis on gelsight sensor: A review,” IEEE Sensors Journal, vol. 20, no. 14, pp. 7628-7638, 2020.
The resolution and quality of the GelSight output (i.e., 3D images) highly depends on its hardware components (e.g., the utilized elastomer, optics, illumination (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) (see S. Dong et al. “Improved gelsight tactile sensor for measuring geometry and slip,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 137-144), fabrication procedure (e.g., thickness and hardness of gel layer (see W. Yuan et al., “Tactile measurement with a gelsight sensor,” Ph.D. dissertation, Massachusetts Institute of Technology, 2014)), and post-processing algorithms (see M. K. Johnson, F. Cole, A. Raj, and E. H. Adelson, “Microgeometry capture using an elastomeric sensor,” ACM Transactions on Graphics (TOG), vol. 30, no. 4, pp. 1-8, 2011). More importantly, the abovementioned parameters also directly affect the sensitivity and the durability of the fabricated VTS, where sensitivity is defined as the ability to obtain high-quality 3D images while applying low interaction forces independent of the shape, size, and material properties of objects, whereas durability refers to the effective life of the VTS without experiencing any wear and tear after multiple use cases on different objects.
The sensitivity and durability of VTSs are highly correlated. To build a sensitive sensor using the fabrication procedure proposed by Yuan et al., durability needs to be often compromised. For instance, reducing the gel layer's stiffness and/or thickness is a common technique to increase the sensitivity of the GelSight sensors; however, this approach may substantially reduce the durability of the sensor when interacting with different objects (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) (see M. K. Johnson et al., F. Cole, A. Raj, and E. H. Adelson, “Microgeometry capture using an elastomeric sensor,” ACM Transactions on Graphics (TOG), vol. 30, no. 4, pp. 1-8, 2011). Moreover, to obtain a high-resolution image using a less sensitive GelSight sensor (i.e., having a high thickness and stiffness gel layer), often a higher interaction force is required to deform the gel layer and obtain high-resolution images. Of note, this might not be feasible for several applications (e.g., high-fidelity manipulation of fragile objects (see N. R. Sinatra et al. “Ultragentle manipulation of delicate structures using a soft robotic gripper,” Science Robotics, vol. 4, no. 33, p. eaax5425, 2019) and surgical applications (see E. Heijnsdijk et al., “Inter- and intraindividual variabilities of perforation forces of human and pig bowel tissue,” Surgical Endoscopy and other interventional Techniques, vol. 17, no. 12, pp. 1923-1926, 2003) (see Y. Liu at al., “Multiphysical analytical modeling and design of a magnetically steerable robotic catheter for treatment of peripheral artery disease,” IEEE/ASME Transactions on Mechatronics, 2022) (see N. Venkatayogi et al., “Classification of colorectal cancer polyps via transfer learning and vision-based tactile sensing,” To be appeared in 2022 IEEE Sensors) and may damage the sensor and reduce its durability. Therefore, there is a critical need for developing a VTS that simultaneously has high sensitivity and durability independent of its application.
Thus, there is a need in the art for systems and methods for quantifying tumor shape, texture, and stiffness to enable universal morphological classification that can be used to predict tumor histology, stage, and survival of colorectal and gastric cancers, and guide appropriate treatment regimens for tumors resulting in improved treatment outcomes. Moreover, endoscopic evaluation with shape and stiffness information using haptics technology would enable early detection of polyps/cancers and morphologic classification that leads to treatment guidance.
Some embodiments of the invention disclosed herein are set forth below, and any combination of these embodiments (or portions thereof) may be made to define another embodiment.
In one aspect, a four-dimensional tactile sensing system comprises a housing including a front-facing camera, and at least one tactile sensor device positioned on an exterior surface of the housing, comprising an elastomer attached to a support plate, a camera positioned proximate to the support plate, and opposite the elastomer, and at least one light source positioned proximate to the support plate and the camera, and opposite the elastomer.
In one embodiment, the housing comprises a pneumatically controlled soft robot. In one embodiment, the housing comprises a cable controlled soft robot. In one embodiment, the housing comprises a pneumatic actuation system configured to actuate the at least one tactile sensor device. In one embodiment, the at least one tactile sensor device is positioned within a skin on the exterior surface of the housing.
In one embodiment, the system further comprises a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, perform the steps of acquiring a set of input images, each input image having an associated known target type and an applied force, performing a principal component analysis on the set of input images to calculate a set of parameters for each image of the set of input images, providing the sets of parameters to a support vector machine to calculate a set of support vectors to classify the parameters, and selecting a subset of the set of support vectors and an applied force threshold, such that for images having an applied force above the applied force threshold, the set of support vectors is configured to predict a target type from the parameters with a characterization confidence of at least 80%.
In one embodiment, the system further comprises a non-transitory computer-readable medium with instructions stored thereon, that when executed by a processor, perform the steps of obtaining a set of calculated support vectors to form a classification scheme, obtaining a set of at least one input image of an object, the at least one input image having an associated applied force value, selecting a subset of the set of at least one input image having an associated applied force value above a threshold, performing a principal component analysis on the subset of input images to calculate a set of parameters of each image in the subset, and applying the classification scheme to the set of parameters to classify the object.
In another aspect, a four-dimensional tactile sensor device comprises an elastomer attached to a support plate, a camera positioned proximate to the support plate, and opposite the elastomer, and at least one light source positioned proximate to the support plate and the camera, and opposite the elastomer.
In one embodiment, the at least one light source is positioned at an oblique angle to the support plate. In one embodiment, the at least one light source comprises a light emitting diode. In one embodiment, the at least one light source comprises a fiberoptic cable. In one embodiment, the at least one light source comprises a first light source of a first color, a second light source of a second color, and a third light source of a third color. In one embodiment, the first color is green, the second color is red, and the third color is blue.
In one embodiment, the elastomer comprises Polydimethylsiloxane (PDMS) or silicone. In one embodiment, the elastomer has a thickness of 0.1 mm to 10 mm. In one embodiment, the elastomer has a hardness of 00-0 to 00-80 or A-10 to A-55. In one embodiment, the elastomer is softer than an object to be measured.
In one embodiment, the support plate comprises a transparent material. In one embodiment, the support plate comprises clear methyl methacrylate. In one embodiment, the support plate has a thickness of 0.1 mm to 10 mm. In one embodiment, the device is configured to measure a four-dimensional morphology of an object comprising a three-dimensional shape of and a stiffness of the object. In one embodiment, the device further comprises a least one marker. In one embodiment, the device further comprises a reflective coating on the surface of the elastomer opposite the support plate.
In another aspect, a four-dimensional tactile morphology method comprises providing at least one tactile sensor device, pressing the at least one tactile sensor device against an object to be measured, and calculating a four-dimensional morphology of the measured object.
In one embodiment, the at least one tactile sensor device comprises an elastomer attached to a support plate, a camera positioned proximate to the support plate, and opposite the elastomer, and at least one light source positioned proximate to the support plate and the camera, and opposite the elastomer.
In one embodiment, the four-dimensional morphology is calculated based on an observation by the camera of a deformation of the elastomer. In one embodiment, the at least one light source is configured to highlight the deformation of the elastomer. In one embodiment, the at least one tactile sensor device is positioned on an exterior surface of a housing. In one embodiment, the four-dimensional morphology of the object comprises a three-dimensional shape of and a stiffness of the object. In one embodiment, the measured object comprises at least one of a tumor, a cancer polyp, and a lesion. In one embodiment, the method further comprises identifying a tumor classification based on the four-dimensional morphology.
In one embodiment, the four-dimensional morphology is calculated using a machine learning or artificial intelligence algorithm. In one embodiment, the machine learning algorithm comprises a convolutional neural network.
In one embodiment, the machine learning or artificial intelligence algorithm is trained via a method comprising acquiring a set of input images, each input image having an associated known target type and an applied force, performing a principal component analysis on the set of input images to calculate a set of parameters for each image of the set of input images, providing the sets of parameters to a support vector machine to calculate a set of support vectors to classify the parameters, and selecting a subset of the set of support vectors and an applied force threshold, such that for images having an applied force above the applied force threshold, the set of support vectors is configured to predict a target type from the parameters with a characterization confidence of at least 80%. In one embodiment, the set of input images comprises interval displacement input images and force interval input images.
In another aspect, a four-dimensional tactile sensing system comprises a flexible sleeve, and at least one tactile sensor device positioned on an exterior surface of the flexible sleeve, comprising an elastomer attached to a support plate, a camera positioned proximate to the support plate, and opposite the elastomer, and at least one light source positioned proximate to the support plate and the camera, and opposite the elastomer.
In another aspect, a method of fabricating the elastomer comprises mixing a multi-part elastomer at a desired mass ratio, molding the elastomer mixture in a mold to form the elastomer, removing the elastomer from the mold, spraying a reflective coating onto the elastomer, and pouring a thin protective coating over the reflective coating.
In one embodiment, the mold is coated to prevent adhesion to the elastomer mixture and to ensure a high elastomer surface quality after molding. In one embodiment, the mold is coated with Ease 200. In one embodiment, the step of molding the elastomer mixture comprises pouring the elastomer mixture into the mold, degassing the mixture in a vacuum chamber, and curing the mixture in a curing station. In one embodiment, the reflective coating has a thickness of 1 μm to 500 μm and comprises a silver coating, a chromium coating, a spray on coating, a specialty mirror effect spray paint, a liquid metal, gallium, or mercury. In one embodiment, the thin protective coating comprises a silicone mixture. In one embodiment, the elastomer has a hardness of 00-18.
In another aspect, an elastomer composition comprises a substrate of Polydimethylsiloxane (PDMS) or silicone, and a reflective coating on a surface of the substrate.
In one embodiment, the elastomer composition has a hardness of 00-0 to 00-80 or A-10 to A-55. In one embodiment, the elastomer composition has a hardness of 00-18. In one embodiment, the reflective coating has a thickness of 1 μm to 500 μm and comprises a silver coating, a chromium coating, a spray on coating, a specialty mirror effect spray paint, a liquid metal, gallium, or mercury. In one embodiment, the substrate further comprises a two part Polydimethylsiloxane (PDMS) or a two part silicone mixture combined with a phenyl trimethicone softener mixed at a mass ratio of 14:10:4.
The foregoing purposes and features, as well as other purposes and features, will become apparent with reference to the description and accompanying figures below, which are included to provide an understanding of the invention and constitute a part of the specification, in which like numerals represent like elements, and in which:
It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clearer comprehension of the present invention, while eliminating, for the purpose of clarity, many other elements found in systems and methods of tactile sensing. Those of ordinary skill in the art may recognize that other elements and/or steps are desirable and/or required in implementing the present invention. However, because such elements and steps are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements and steps is not provided herein. The disclosure herein is directed to all such variations and modifications to such elements and methods known to those skilled in the art.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, exemplary methods and materials are described.
As used herein, each of the following terms has the meaning associated with it in this section.
The articles “a” and “an” are used herein to refer to one or to more than one (i.e., to at least one) of the grammatical object of the article. By way of example, “an element” means one element or more than one element.
“About” as used herein when referring to a measurable value such as an amount, a temporal duration, and the like, is meant to encompass variations of ±20%, ±10%, ±5%, ±1%, and ±0.1% from the specified value, as such variations are appropriate.
Ranges: throughout this disclosure, various aspects of the invention can be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Where appropriate, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 2.7, 3, 4, 5, 5.3, and 6. This applies regardless of the breadth of the range.
Referring now in detail to the drawings, in which like reference numerals indicate like parts or elements throughout the several views, in various embodiments, presented herein is a tactile sensing system, device and method.
In some embodiments, the system 100 can provide haptic feedback. In some embodiments, the system 100 can map the 3D shape and stiffness of small features of a measured object 125, up to 100 μm. In some embodiments, the system 100 can be utilized to create a topological mapping of internal anatomies and create high resolution detailed mapping of the internal surface of internal anatomies such as the colon or stomach, for example. In some embodiments the tactile sensing device 150 can have a normal displacement via a pneumatic actuation to provide normal interaction forces for texture and stiffness measurement, wherein only tactile sensing device 150 is displaced rather than the whole robot 105. In some embodiments, actuation of the embedded tactile sensing device 150 can be performed via a cable actuator and/or a pneumatic actuator configured to move the tactile sensing robot system 100. In some embodiments, actuation of the embedded tactile sensing device 150 can be performed via a cable actuator and/or a pneumatic actuator configured to move the embedded tactile sensing device 150 independently of the tactile sensing robot system 100.
Referring now to
In some embodiments, the at least one light source 160 comprises a first light source of a first color, a second light source of a second color, and a third light source of a third color. In some embodiments, the first color is green, the second color is red, and the third color is blue, but any suitable number and combination of colors can be utilized. In some embodiments, the first, second and third colors each comprise at least one of visible light, ultraviolet (UV) light, infrared (IR) light, and any other suitable light or combination thereof. The at least one light source 160 can be configured to simultaneously illuminate the tactile sensor device 150 or can be programmed to provide application dependent illumination where a subset of the at least one light source 160 is used to illuminate the tactile sensor device 150 for a set amount of time.
In some embodiments, the elastomer 170 comprises Polydimethylsiloxane (PDMS), silicone, or any type of flexible elastomer, has a thickness of 0.1 mm to 10 mm, and has a hardness of 00-0 to 00-80 or A-10 to A-55. In some embodiments, the elastomer 170 is softer than an object to be measured 125.
In some embodiments, the support plate 165 comprises a transparent material. In some embodiments, the support plate comprises clear methyl methacrylate plate (acrylic plate) or any other suitable material, and has a thickness of 0.1 mm to 10 mm.
In some embodiments, the device 150 further includes a reflective coating 175 on a surface of the elastomer 170, such as the surface opposite the support plate 165. In some embodiments, the reflective coating 175 is configured to enhance light collection. In some embodiments, the reflective coating 175 is used to create a mirror effect. In some embodiments, the reflective coating 175 comprises a silver coating, a chromium coating, a spray on coating, a specialty mirror effect spray paint, a liquid metal such as gallium or mercury, or any suitable reflective coating or combination thereof. In some embodiments, the reflective coating 175 has a thickness of 1 μm to 500 μm or any other suitable thickness.
In some embodiments, the device 150 further includes at least one marker on a surface of and/or within the elastomer 170. In some embodiments, the at least one marker 180 is configured for use as a reference mark for vision-based calculations of properties of a sample being measured with the device 150 such as stiffness, size, and shape, for example.
Any suitable number of tactile sensor devices 150, spacing between the devices 150, and arrangement of the devices 150 can be utilized. In some embodiments, the tactile sensor devices 150 can be arranged linearly, annularly, spirally, geometrically, or in any other suitable arrangement or combination thereof. In some embodiments, the center to center spacing of the tactile sensor devices 150 is 5 mm to 100 mm. The tactile sensor device 150 can be utilized to measure fine textural details, size, shape and stiffness an object of interest. In some embodiments, the object of interest can be a polyp or lesion. In some embodiments, the tactile sensing device 150 is configured to detect texture-based features.
In some embodiments, the system 200 can provide haptic feedback. In some embodiments, the system 200 can map the 3D shape and stiffness of small features of a measured object, up to 100 μm. In some embodiments, the system 200 can be utilized to create a topological mapping of internal anatomies and create high resolution detailed mapping of the internal surface of internal anatomies such as the colon or stomach, for example.
At Operation 310, the at least one tactile sensor device 150 is pressed against an object to be measured 125. As the device 150 is pressed against the object to be measured 125, the elastomer 170 is deformed.
The method ends at Operation 315, where a four-dimensional morphology of the measured object 125 is calculated. The four-dimensional morphology is calculated based on an observation by the camera 155 of a deformation of the elastomer 170. In some embodiments, the four-dimensional morphology of the object 125 comprises the three-dimensional shape of and the stiffness of the object. In some embodiments, the at least one light source 160 is configured to highlight the deformation of the elastomer 170.
In some embodiments, the measured object 125 comprises at least one of a tumor, a cancer polyp, and a lesion. In some embodiments, the method 300 further comprises identifying a tumor classification based on the four-dimensional morphology. In some embodiments, the four-dimensional morphology is calculated using a machine learning algorithm. In some embodiments, the machine learning algorithm comprises a convolutional neural network. In some embodiments, the algorithm is configured to provide real time tumor identification, tumor classification and/or stiffness scoring based on the visual feedback of the tactile sensor device 150.
In some embodiments, the measure object 125 comprises an interior component of a pipe in an industrial or municipal system. For example, the system 100 can be utilized in a pipe inspection setting taking measurements of pipe connection points to look for degradation and/or cracking. In some embodiments, the measured object comprises a fruit. For example, the system 100 can be utilized in a farming application such as harvesting by taking a measurement of the fruit to see if is ripe enough to pick.
In another aspect, a method of fabricating the elastomer comprises mixing a multi-part elastomer at a desired mass ratio, molding the elastomer mixture in a mold to form the elastomer, removing the elastomer from the mold, spraying a reflective coating onto the elastomer, and pouring a thin protective coating over the reflective coating.
In one embodiment, the mold is coated to prevent adhesion to the elastomer mixture and to ensure a high elastomer surface quality after molding. In one embodiment, the mold is coated with Ease 200. In one embodiment, the step of molding the elastomer mixture comprises pouring the elastomer mixture into the mold, degassing the mixture in a vacuum chamber, and curing the mixture in a curing station. In one embodiment, the reflective coating comprises sprayed specialty mirror effect spray paint. In one embodiment, the thin protective coating comprises a silicone mixture. In one embodiment, the elastomer has a hardness of 00-18. In some embodiments, markers are place in or on the elastomer during the molding step.
In some embodiments an elastomer composition comprises a substrate of Polydimethylsiloxane (PDMS) or silicone, and a reflective coating on a surface of the substrate. In one embodiment, the elastomer composition has a hardness of 00-0 to 00-80 or A-10 to A-55. In one embodiment, the elastomer composition has a hardness of 00-18. In one embodiment, the reflective coating has a thickness of 1 μm to 500 μm and comprises a silver coating, a chromium coating, a spray on coating, a specialty mirror effect spray paint, a liquid metal, gallium, or mercury. In one embodiment, the substrate comprises a two part Polydimethylsiloxane (PDMS) or a two part silicone mixture combined with a phenyl trimethicone softener mixed at a mass ratio of 14:10:4.
In some aspects of the present invention, software executing the instructions provided herein may be stored on a non-transitory computer-readable medium, wherein the software performs some or all of the steps of the present invention when executed on a processor.
Aspects of the invention relate to algorithms executed in computer software. Though certain embodiments may be described as written in particular programming languages, or executed on particular operating systems or computing platforms, it is understood that the system and method of the present invention is not limited to any particular computing language, platform, or combination thereof. Software executing the algorithms described herein may be written in any programming language known in the art, compiled or interpreted, including but not limited to C, C++, C#, Objective-C, Java, JavaScript, MATLAB, Python, PHP, Perl, Ruby, or Visual Basic. It is further understood that elements of the present invention may be executed on any acceptable computing platform, including but not limited to a server, a cloud instance, a workstation, a thin client, a mobile device, an embedded microcontroller, a television, or any other suitable computing device known in the art.
Parts of this invention are described as software running on a computing device. Though software described herein may be disclosed as operating on one particular computing device (e.g. a dedicated server or a workstation), it is understood in the art that software is intrinsically portable and that most software running on a dedicated server may also be run, for the purposes of the present invention, on any of a wide range of devices including desktop or mobile devices, laptops, tablets, smartphones, watches, wearable electronics or other wireless digital/cellular phones, televisions, cloud instances, embedded microcontrollers, thin client devices, or any other suitable computing device known in the art.
Similarly, parts of this invention are described as communicating over a variety of wireless or wired computer networks. For the purposes of this invention, the words “network”, “networked”, and “networking” are understood to encompass wired Ethernet, fiber optic connections, wireless connections including any of the various 802.11 standards, cellular WAN infrastructures such as 3G, 4G/LTE, or 5G networks, Bluetooth®, Bluetooth® Low Energy (BLE) or Zigbee® communication links, or any other method by which one electronic device is capable of communicating with another. In some embodiments, elements of the networked portion of the invention may be implemented over a Virtual Private Network (VPN).
Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
The storage device 420 is connected to the CPU 450 through a storage controller (not shown) connected to the bus 435. The storage device 420 and its associated computer-readable media, provide non-volatile storage for the computer 400. Although the description of computer-readable media contained herein refers to a storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the computer 400.
By way of example, and not to be limiting, computer-readable media may comprise computer storage media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by the computer.
According to various embodiments of the invention, the computer 400 may operate in a networked environment using logical connections to remote computers through a network 440, such as TCP/IP network such as the Internet or an intranet. The computer 400 may connect to the network 440 through a network interface unit 445 connected to the bus 435. It should be appreciated that the network interface unit 445 may also be utilized to connect to other types of networks and remote computer systems.
The computer 400 may also include an input/output controller 455 for receiving and processing input from a number of input/output devices 460, including a keyboard, a mouse, a touchscreen, a camera, a microphone, a controller, a joystick, or other type of input device. Similarly, the input/output controller 455 may provide output to a display screen, a printer, a speaker, or other type of output device. The computer 400 can connect to the input/output device 460 via a wired connection including, but not limited to, fiber optic, ethernet, or copper wire or wireless means including, but not limited to, Bluetooth, Near-Field Communication (NFC), infrared, or other suitable wired or wireless connections.
As mentioned briefly above, a number of program modules and data files may be stored in the storage device 420 and RAM 410 of the computer 400, including an operating system 425 suitable for controlling the operation of a networked computer. The storage device 420 and RAM 410 may also store one or more applications/programs 430. In particular, the storage device 420 and RAM 410 may store an application/program 430 for providing a variety of functionalities to a user. For instance, the application/program 430 may comprise many types of programs such as a word processing application, a spreadsheet application, a desktop publishing application, a database application, a gaming application, internet browsing application, electronic mail application, messaging application, and the like. According to an embodiment of the present invention, the application/program 430 comprises a multiple functionality software application for providing word processing functionality, slide presentation functionality, spreadsheet functionality, database functionality and the like.
The computer 400 in some embodiments can include a variety of sensors 465 for monitoring the environment surrounding and the environment internal to the computer 400. These sensors 465 can include a Global Positioning System (GPS) sensor, a photosensitive sensor, a gyroscope, a magnetometer, thermometer, a proximity sensor, an accelerometer, a microphone, biometric sensor, barometer, humidity sensor, radiation sensor, or any other suitable sensor.
In some embodiments, artificial intelligence (AI) and machine learning (ML) methods and algorithms are utilized for performing image recognition, calculation, and classification of the measured objects based on the measured morphology. In some embodiments, the machine learning algorithm comprises a convolutional neural network, or any other suitable machine learning algorithm. In some embodiments, the machine learning or artificial intelligence algorithm is trained via a method comprising acquiring a set of input images, each input image having an associated known target type and an applied force, performing a principal component analysis on the set of input images to calculate a set of parameters for each image of the set of input images, providing the sets of parameters to a support vector machine to calculate a set of support vectors to classify the parameters, and selecting a subset of the set of support vectors and an applied force threshold, such that for images having an applied force above the applied force threshold, the set of support vectors is configured to predict a target type from the parameters with a characterization confidence of at least 80%. In some embodiments, the machine learning or artificial intelligence algorithm calculates the four-dimensional morphology via a method comprising obtaining a set of calculated support vectors to form a classification scheme, obtaining a set of at least one input image of an object, the at least one input image having an associated applied force value selecting a subset of the set of at least one input image having an associated applied force value above a threshold, performing a principal component analysis on the subset of input images to calculate a set of parameters of each image in the subset, and applying the classification scheme to the set of parameters to classify the object.
The invention is now described with reference to the following Examples. These Examples are provided for the purpose of illustration only and the invention should in no way be construed as being limited to these Examples, but rather should be construed to encompass any and all variations which become evident as a result of the teaching provided herein.
Without further description, it is believed that one of ordinary skill in the art can, using the preceding description and the following illustrative examples, make and utilize the present invention and practice the claimed methods. The following working examples therefore specifically point out exemplary embodiments of the present invention and are not to be construed as limiting in any way the remainder of the disclosure.
Since colonoscopic environments are spatially constrained, design optimization is necessary to reduce the size of the tactile sensor device 150 while maximizing the sensorized tactile sensing region to provide a sufficient contact surface (i.e., maximum size of 10 mm×10 mm) between the sensor and polyp's surface. To this end, the required size for the housing body of the tactile sensor device 150 was analyzed as well as angle of view and focal length of camera. Based on the robot geometry and dimensions used for the experiment, the maximum allowable housing body before interfering with tendon routes was calculated as 20 mm by 20 mm. To allow sufficient room for LEDs and wiring, the maximum active sensorized surface was then obtained as 15 mm by 15 mm. The focal length, given angle of view (AOV), determined the field of view. Based on the performed analysis and calculations, a compact camera was selected for the experimental application (Noir Mini Camera, Arducam), which had AOV of 64 by 48 degrees. The field of view of the camera was calculated as a function of focal length from the geometrical relationships of the camera optics. Based on this relationship, the desired focal length was found to be 14.1 mm to achieve the desired field of view (i.e., 15 mm by 11.5 mm) defined by robot's space constraints. In order to embed the sensor within the developed soft robot's body, external geometry of the sensor's housing body was designed such that it fills the considered cut in the soft robot's molded geometry. The sensor was then embedded into the soft robot's body sing a silicone adhesive (Sil-Poxy, SmoothOn, Inc.).
In contrast to previously presented polyp phantoms in the literature that largely simplified polyp phantoms to generic lumps and simple geometrical structures, the anatomically correct texture and fine details of polyps was replicated. To represent the widely varying polyp classes, a subset including the slightly elevated nonpolypoid polyp (IIa), slightly depressed nonpolypoid polyp (IIc), pedunculated polyp (Ip) and laterally spreading type (LST), were selected for experimentation. Aside from the size and geometry of these polyps, to simulate different stiffnesses for these phantoms, they were fabricated from the following materials and with different high resolution printing technologies (i.e., SLA and Polyjet 3D Printing): Clear Resin (FLGPCL04, Formlabs Inc.), Vero (VeroBlackPlus, Stratasys, Ltd.), Elastic Resin (FLELCL01, Formlabs Inc.) and Tango (TangoBlackPlus, Stratasys, Ltd.). The properties of the polyp phantom's materials are listed in the table of
The clinical images of the selected polyp types, the CAD designs and their corresponding 3D printed models, as well as their tactile sensor device representation at 3.0 N of applied force are shown in
The result shown in
To explore the relationship between applied force and characterization confidence, classification and classification confidence of the images were studied. To this end, a support vector machine (SVM) approach was utilized. Specifically, to avoid overfitting based on the relatively limited image set, a cubic polynomial kernel was selected, and the regularization parameter was tuned. The model was trained using a random sample of the images from the displacement-normal force experiments and tested on the device images from the force increment experiments (i.e., 0.0-3.0 N at 0.5 N increments). Then, the distances to the support vectors for each image was calculated. The distances were then normalized to get a prediction confidence where the sum of the prediction confidences for each polyp type on an image summed to 100%. The resulting prediction confidences for each force increment and for each polyp type were plotted in
To address the sensitivity and durability trade-off of common VTSs, further disclosed are embodiments of designs and fabrication methods of a hyper-sensitive and high-fidelity VTS (referred to as HySenSe) that requires a very low interaction force (i.e., <1.5N) to obtain high-resolution images. To fabricate the high-fidelity VTS, the standard fabrication procedure of GelSight sensors (see W. Yuan et al., “Tactile measurement with a gelsight sensor,” Ph.D. dissertation, Massachusetts Institute of Technology, 2014) was followed and altered to drastically improve the device's sensitivity and obtain high-quality images, all while applying a very low interaction force that does not compromise its durability. To thoroughly evaluate the performance of HySenSe, 3D image outputs were analyze and compared its results with a similar Gelsight sensor on different objects (with different shapes, textures, and stiffness) and under various interaction forces.
As illustrated in
To thoroughly evaluate the performance of HySenSe, similar GelSight and HySenSe sensors were fabricated. For both sensors, a 5 MP camera (Arducam ¼ inch 5 MP sensor mini) was used that was fixed to the rigid frame printed with high-resolution printing technology, a Form 3 printer (Formlabs Form 3, Formlabs Inc.), and the clear resin material (FLGPCL04, Formlabs Inc.). The rigid frame height was designed as hs=24 mm, determined based on the camera focus and field of view. Of note, the other dimensions of the rigid frame are determined based on the size of the gel layer described below. Moreover, an array of Red, Green and Blue LEDs (WL-SMTD Mono-Color 150141RS63130, 150141GS63130, 150141BS63130, respectively) were placed and arranged 120 degrees apart. To have identical fabricated gel layers for these sensors and in order to build samples with identical volume and geometry, the volume of the spherical shape gel layers V was calculated as follows:
where, as conceptually demonstrated in
As mentioned, the sensitivity of a VTS can be controlled using the utilized hardware components (see W. Yuan et al., “Tactile measurement with a gelsight sensor,” Ph.D. dissertation, Massachusetts Institute of Technology, 2014) and/or post processing algorithms (see M. K. Johnson, F. Cole, A. Raj, and E. H. Adelson, “Microgeometry capture using an elastomeric sensor,” ACM Transactions on Graphics (TOG), vol. 30, no. 4, pp. 1-8, 2011). Nevertheless, in this study, to fabricate a hyper-sensitive and durable VTS, it is shown that solely by changing one step during fabrication of the GelSight's gel layer, one can substantially increase the device's sensitivity and durability. The following briefly describes and compares the steps for fabricating a gel layer for GelSight sensor and the modified procedure for fabricating the gel layer for the HySenSe sensor.
Step 1: For the fabrication of the gel layer (shown in
Step 2: After curing, the matte-colored aluminum powder (AL-101, Atlantic Equipment Engineers) was brushed on the gel layer's dome surface to avoid light leakage. Of note, the fabricated gel layer had ws=33.8 mm and ts=4.5 mm, which relatively has a higher thickness compared to the previous literature (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) (see M. K. Johnson et al., “Microgeometry capture using an elastomeric sensor,” ACM Transactions on Graphics (TOG), vol. 30, no. 4, pp. 1-8, 2011). Remarkably, these parameters can readily be changed depending on the application requirements.
Step 3: Finally, a thin layer of silicone with the addition of grey pigment (blend of both black and white pigments-Silc Pig Black, Silc Pig White, Smooth-On Inc) was poured, with identical proportion described in STEP I, on the surface of gel layer to prevent light leakage and stabilize the aluminum powder layer. Notably, the hardness of the fabricated gel layer was measured as 00-20 using a Shore 00 scale durometer (Model 1600 Dial Shore 00, Rex Gauge Company). The fabricated gel layer is shown in
To fabricate the gel layer for the HySenSe sensor (shown in
To thoroughly compare the sensitivity and durability of the HySenSe and GelSight sensors independent of the size, shape, thickness, texture, and hardness of objects, as shown in
To summarize, to improve the resolution of the VTS image outputs various approaches such as increasing the interaction forces (see W. Yuan et al., “Shape-independent hardness estimation using deep learning and a gelsight tactile sensor,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 951-958), reducing the hardness and thickness of gel layer, complex fabrication procedures (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017), and post processing image processing algorithms (see M. K. Johnson et al., “Microgeometry capture using an elastomeric sensor,” ACM Transactions on Graphics (TOG), vol. 30, no. 4, pp. 1-8, 2011) have been implemented and proposed in the literature. Nevertheless, as demonstrated herein, by solely changing one step (i.e., Step 2) during the fabrication procedure of the gel layer, the sensitivity of the GelSight sensor can be drastically improved. More specifically, by using a mirror spray paint instead of aluminum powder and grey pigments, not only can one improve the reflectivity of the illumination, but one can also reduce the thickness of the coating to substantially improve the sensitivity.
Aside from sensitivity, the disclosed fabrication procedure for the HySenSe also can resolve the existing sensitivity and durability trade-off in VTSs. As described, to improve the durability of the GelSight sensors, the thickness and/or the hardness of the gel layer needs to be increased while this may drastically deteriorate the sensitivity of these sensors (see S. Dong et al., “Improved gelsight tactile sensor for measuring geometry and slip,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 137-144). The typical remedy for such situations is to increase the interaction force between the object and the gel layer to obtain a high-quality image, which may not be feasible for many applications and may damage the sensor, too. Nevertheless, as shown in
Disclosed embodiments were also utilized toward collectively addressing the abovementioned limitations of existing VTSs in quantitative measurements of gel layer deformation. Further explored was a design, fabrication, and characterization of a novel Quantitative Vision-based Tactile Sensor (Q-VTS). The core of the disclosed sensor is the utilization of miniature 1.5 mm×1.5 mm synthetic square markers with inner binary patterns and a broad black border called ArUco Markers (see S. Garrido-Jurado, “Automatic generation and detection of highly reliable' fiducial markers under occlusion,” Pattern Recognition, vol. 47, no. 6, pp. 2280-2292, 2014). Each ArUco marker can provide real time camera pose estimation that can be used as a quantitative measure for obtaining deformation of the Q-VTS gel layer. Moreover, thanks to the use of ArUco markers, a novel fabrication procedure that mitigates the challenges mentioned above during the fabrication of VTSs is used. Particularly, the disclosed fabrication facilitates the integration and adherence of markers with the gel layer to robustly and reliably obtain a quantitative measure of deformation in real-time.
Pose estimation is a computer vision problem that determines the orientation and position of the camera with respect to a given object and has great importance in many computer vision applications ranging from surgical robotics (see F. P. Villani et al., Development of an augmented reality system based on marker tracking for robotic assisted minimally invasive spine surgery,” in International Conference on Pattern Recognition. Springer, 2021, pp. 461-475), and augmented reality (see C. Mela et al., “Novel multimodal, multiscale imaging system with augmented reality,” Diagnostics, vol. 11, no. 3, p. 441, 2021), to robot localization (see A. de Oliveira Junior et al., “Improving the mobile robots indoor localization system by combining slam with fiducial markers,” in 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education (WRE). IEEE, 2021, pp. 234-239). Binary square fiducial markers have been emerged as an accurate and reliable solution for the pose estimation problem in various applications (see S. Garrido-Jurado et al., “Automatic generation and detection of highly reliable' fiducial markers under occlusion,” Pattern Recognition, vol. 47, no. 6, pp. 2280-2292, 2014). These markers offer easily discernible patterns with strong visual characteristics through their four corners and specific ID to obtain the camera pose. Moreover, their unique inner binary codes add robustness for the misdetections and reduce false positive detections (see M. Kalaitzakis et al., “Fiducial markers for pose estimation,” Journal of Intelligent & Robotic Systems, vol. 101, no. 4, pp. 1-26, 2021). Various fiducial marker libraries has been developed progressively in the literature to address the pose estimation problem such as AprilTag (see E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in 2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 3400-3407), ARTag (see M. Fiala, “Artag, a fiducial marker system using digital techniques,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR′05), vol. 2. IEEE, 2005, pp. 590-596), and ArUco (see S. Garrido-Jurado et al., “Automatic generation and detection of highly reliable' fiducial markers under occlusion,” Pattern Recognition, vol. 47, no. 6, pp. 2280-2292, 2014). A detailed review of fiducial marker packages and descriptions can be found in C. Mela et al., “Novel multimodal, multiscale imaging system with augmented reality,” Diagnostics, vol. 11, no. 3, p. 441, 2021. In this work, ArUco markers were used for estimating the deformation of the VTS gel layer, as they can provide a high detection rate in real-time and allow the use of reconfigurable libraries with less computing time.
Working Principle and Constructing Elements of QVTS: As demonstrated in
Fabrication Procedure of the Q-VTS: To fabricate the Q-VTS, a 13 MP autofocus USB camera (IEights 4K ⅓ inch with an IMX 415 sensor and 5-50 mm varifocal lens) was used that was fixed to the rigid frame printed with a 3D printer (E2, Raise3D), and the PLA filament (Art White Extreme Smooth Surface Quality, Raise3D). An autofocus camera was a suitable selection for Q-VTS as one could control and optimize the focal distance through the deformation procedure of the gel layer and always ensure a clear output image. In other words, a fixed focal length camera may create blurry visuals after exceeding the focus threshold and throughout the gel layer deformation. The rigid frame height was designed as hs=55 mm based on the camera focus and 100° field of view. Of note, the zoom parameter of the autofocus camera and the distance between the camera and the ArUco markers were optimized to find the balance between the detection rate and the correct pose estimation of markers.
Fabrication Procedure of the Gel Layer: The following paragraphs describe and compare the steps taken for fabricating a typical GelSight sensor and the novel fabrication method for the Q-VTS sensor.
GelSight Sensor: STEP 1: To fabricate the deformable gel layer (as illustrated in
GelSight Sensor: STEP 2: After the curing step, black marker dots on the sensor surface could be attached either using waterslide decal paper (see W. Yuan et al., “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” Sensors, vol. 17, no. 12, p. 2762, 2017) or manually marking by hand (see W. Yuan et al., “Shape-independent hardness estimation using deep learning and a gelsight tactile sensor,” in 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017, pp. 951-958). For the first option, the marker dot pattern was printed on the glossy side of the water transfer paper via a laserjet printer. Then, the transfer paper was soaked in medium-temperature water to wet the paper surface to maneuver and peel the printed side off easily. Afterwards, the transfer paper was placed on the dome-shape gel layer with the marker dots facing up while separating the backing paper. Of note, this arduous procedure demands multiple repetitions and requires experience in working with the decal papers for the integration of it on the sensor surface. Even if the transfer paper is placed correctly on the surface of the gel layer, it will most likely be wrinkled when it interacts with an object, and therefore deteriorates the sensor sensitivity and quality of the output images. The second option is relatively manageable but lacks a consistent black dot marking procedure and does not provide a quantitative measure for the gel layer deformation.
GelSight Sensor: STEP 3: This step includes covering the printed markers on the gel layer's surface. To this end, first, the matte-colored aluminum powder (AL-101, Atlantic Equipment Engineers) was brushed on the gel layer's dome surface to avoid light leakage. Finally, a thin layer of silicone with the addition of grey pigment (blend of both black and white pigments-Silc Pig Black, Silc Pig White, Smooth-On Inc) was poured, with identical proportion described in STEP 1, on the surface of gel layer to stabilize the aluminum powder layer and prevent light leakage since there exist RGB LEDs within the rigid casing. Notably, the hardness of the gel layer sample was measured as 00-20 using a Shore 00 scale durometer (Model 1600 Dial Shore 00, Rex Gauge Company).
Q-VTS: To fabricate the deformable gel layer for the novel sensor, above-described procedure in STEP 1 was followed. The significant change in the Q-VTS fabrication procedure occurs in STEP 2 and STEP 3 in which, instead of utilizing black dot marker patterns, 25 square ArUco Markers were used with the size of 1.5 mm×1.5 mm. They were adhered separately and one by one to the Q-VTS gel layer surface. Each ArUco Markers were printed on a water transfer paper (Sunnyscopa) using a laserjet printer (Color Laser Jet Pro MFP M281fdw, HewlettPackard) with 600×600 DPI to obtain the best printing quality from the utilized printer. Of note, the 1.5 mm×1.5 mm marker size was determined after performing a few preliminary tests with the detection algorithm. It is worth mentioning that high DPI printing quality would enable using smaller marker sizes while still having a high detection rate. Before placing each ArUco marker, the sensor's surface was brushed with the versatile decal setting solution, Micro Set (Microscale Industries), to increase the adhesion and prepare the surface for the application of the transfer paper. After 510 minutes, each marker was placed with precision tweezers one by one by following the instructional procedures of the transfer paper to create a 5×5 array on the sensor surface. Of note, as each marker was positioned separately, and ArUco Markers could be detected independently, regardless of their positioning, orientation, and uniformity, the problems mentioned earlier in STEP 2 of GelSight preparation were not an issue.
It is worth noting that due to the use of ambient light instead of LEDs, the disclosed fabrication method eliminates the need for additional aluminum powder brushing in STEP 3. Thus, as the last fabrication step, a thin layer of silicone mixture with the addition of white pigment (Silc Pig White, Smooth-On Inc) and the same proportion as in STEP 1, was poured on the sensor surface to cover the ArUco Markers. Of note, white pigment is preferred to easily distinguish the black-colored patterns of the markers from the white background and aid the computer vision algorithm during the detection procedure.
As demonstrated in
A 4×4 bit library was selected with 50 ArUco markers to have a robust detection of errors for the 25 ArUco markers (as demonstrated in
After integrating each of the 25 ArUco Marker on the deformable gel layer (shown in
As shown, the experimental setup included of the Q-VTS, a 3D printed flat square object designed for testing the Q-VTS deformation measurement, a single-row linear stage with 1 μm precision (M-UMR12.40, Newport) for the precise control of the flat square plate displacement, a digital force gauge with 0.02 N resolution (Mark-10 Series 5, Mark10 Corporation) attached to the linear stage to track the interaction force between the flat square plate and Q-VTS, and a Dell Latitude 5400 for streaming and recording the video for data processing. Spyder, the Scientific Python Development Software, was used to complete the camera calibration and process the acquired ArUco Marker data for the pose estimation.
In order to evaluate the performance of the Q-VTS in detecting the ArUco markers attached to the sensor surface and the pose estimation, a camera calibration was performed using OpenCV. Of note, camera calibration was one of the most essential steps to identify markers correctly and determine each marker's accurate orientation and distance vectors relying on 10×7 sized checkerboard that was patterned with 1.5 mm×1.5 mm squares. A set of 36 checkerboard images was used that were captured from different orientations and distances. After processing these checkerboard visuals, a 3×3 camera intrinsic matrix and radial and tangential distortion coefficients were obtained. Notably, a camera intrinsic matrix (CIM) is a unique matrix specific to a camera, including both focal length (fx,fy) and optical centers (cx and cy) (see M. Beyeler, OpenCV with Python blueprints. Packt Publishing Ltd, 2015). It is expressed as a 3×3 matrix as CIM=[fx 0 cx; 0 fy cy; 0 0 1].
After the acquisition of both the CIM and distortion coefficients as an .yaml file, a 3D printed flat square plate was attached to the force gauge using the threaded connection at the base. Then, Q-VTS was fixed to the optical table to prevent any undesired slipping or sliding. Next, the linear stage was precisely moved until the square plate contacted the Q-VTS. Notably, a force gauge was placed on the linear stage to detect the initial touch between the square plate and Q-VTS to ensure that there was no deformation during the initial positioning. After arranging the hardware for measurements, the main detection and pose estimation algorithm was initialized to screen frame numbers, recognized marker IDs in real-time, and record numerical data involving detection rate, pose, and the orientation of each ArUco marker to the excel file. During the characterization experiment, a square flat plate was driven 0.4 mm per 200 frames until 2 mm displacement to record the pose for each displacement.
In summary, a vision based tactile sensor (called Q-VTS) was designed and built to address the limitations of conventional VTSs, including the time-consuming and arduous fabrication methods for the marker attachment, inconsistent marking patterns and wrinkling problems that deteriorate the quality of the tactile information extraction, and lack of quantitative deformation evaluation of typical VTSs. Thanks to the use of ArUco markers enables an accurate estimation of the gel layer deformation in X, Y, and Z directions regardless of the placement of ArUco markers. The performance and efficacy of Q-VTS in estimating the deformation of the sensor's gel layer were experimentally evaluated and verified. An accurate estimation of the deformation of the gel layer with a low relative error of <5% in the Z direction and less than 0.5 mm in both the X and Y direction was achieved.
The following references are incorporated herein by reference in their entirety:
The disclosures of each and every patent, patent application, and publication cited herein are hereby incorporated herein by reference in their entirety. While this invention has been disclosed with reference to specific embodiments, it is apparent that other embodiments and variations of this invention may be devised by others skilled in the art without departing from the true spirit and scope of the invention.
This application claims priority to U.S. provisional application No. 63/276,259 filed on Nov. 5, 2021, incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2022/048940 | 11/4/2022 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63276259 | Nov 2021 | US |
