Patent Application
20250134387
This present disclosure generally relates to computer imaging, tissue characterization, computer vision, and/or to the field of medical imaging, particularly to devices/apparatuses, systems, methods, and storage mediums for using artificial intelligence (“AI”) for performing tissue characterization in one or more images and/or for using one or more imaging modalities, including but not limited to, angiography, Optical Coherence Tomography (OCT), Multi-modality OCT (MM-OCT), near-infrared fluorescence (NIRF), near-infrared auto-fluorescence (NIRAF), OCT-NIRAF, robot imaging, robot imaging, continuum robot imaging, etc. Examples of OCT applications include imaging, evaluating, and diagnosing biological objects, including, but not limited to, for gastro-intestinal, cardio, and/or ophthalmic applications, and being obtained via one or more optical instruments, including, but not limited to, one or more optical probes, one or more catheters, one or more endoscopes, one or more capsules, and one or more needles (e.g., a biopsy needle). One or more devices, systems, methods and storage mediums for characterizing, examining and/or diagnosing, and/or measuring viscosity of, a sample or object (e.g., tissue, an organ, a portion of a patient, etc.) in artificial intelligence application(s) using an apparatus or system that uses and/or controls one or more imaging modalities are discussed herein.
Fiber optic catheters and endoscopes have been developed to access to internal organs. For example, in cardiology, OCT has been developed to see (e.g., capture and visualize) depth resolved images of vessels with a catheter. The catheter, which may include a sheath, a coil and an optical probe, may be navigated to a coronary artery.
Optical coherence tomography (OCT) is a technique for obtaining high-resolution cross-sectional images of tissues or materials, and OCT enables real time visualization. The aim of the OCT techniques is to measure the time delay of light by using an interference optical system or interferometry, such as via Fourier Transform or Michelson interferometers. A light from a light source delivers and splits into a reference arm and a sample (or measurement) arm with a splitter (e.g., a beamsplitter). A reference beam is reflected from a reference mirror (partially reflecting or other reflecting element) in the reference arm while a sample beam is reflected or scattered from a sample in the sample arm. Both beams combine (or are recombined) at the splitter and generate interference patterns. The output of the interferometer is detected with one or more detectors, such as, but not limited to, photodiodes or multi-array cameras, in one or more devices, such as, but not limited to, a spectrometer (e.g., a Fourier Transform infrared spectrometer). The interference patterns are generated when the path length of the sample arm matches that of the reference arm to within the coherence length of the light source. By evaluating the output beam, a spectrum of an input radiation may be derived as a function of frequency. The frequency of the interference patterns corresponds to the distance between the sample arm and the reference arm. The higher frequencies are, the greater are the differences in path length. Single mode fibers may be used for OCT optical probes, and double clad fibers may be used for fluorescence and/or spectroscopy. A multi-modality system, such as, but not limited to, an OCT, fluorescence, and/or spectroscopy system with an optical probe, is developed to obtain multiple information at the same time.
OCT also allows the evaluation of luminal dimensions and the assessment of vessel wall morphology. As discussed in Li L and Jia T, Optical Coherence Tomography Vulnerable Plaque Segmentation Based on Deep Residual U-Net, Reviews in Cardiovascular medicine, September 2019, the disclosure of which is incorporated by reference herein in its entirety, mainly four different tissue types may be detected in OCT: (1) calcium (appears as a signal poor region with sharply delineated borders), (2) fibrous tissue (appears as a high back-scattering homogeneous area), (3) lipid tissue (appears as a signal poor region with diffused borders or no back-scattering signal), and (4) mixed tissue (appears as a heterogeneous area having characteristics from multiple tissue types).
Traditionally, OCT and plaque characterization is performed manually, a process which is time consuming and has high inter-observer variability due to inexperienced operators.
Additionally, attempts at detecting a lumen and/or a vessel wall and at characterizing tissue using segmentation and decision trees have been performed. However, such a method is time consuming since the method requires every cross sectional OCT frame of the pullback to be processed, and the method does not take into consideration spatial connection of tissue in adjacent frames. Use of a deep learning network to characterize tissue by processing every two-dimensional (2D) frame of a pullback also increases computational time and hardware cost since a dedicated processor or a graphics processing unit (GPU) is needed. Furthermore, characterization of tissue using the above techniques is often dependent on lumen and vessel wall detection, and this limits the accuracy of the tissue characterization when a lumen is not well detected.
Accordingly, it would be desirable to provide at least one imaging or optical apparatus/device, system, method, and storage medium that applies machine learning and/or deep learning to evaluate and characterize a target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) with a higher success rate when compared to traditional techniques, and to use the one or more results to characterize the target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) more efficiently. It also would be desirable to provide one or more probe/catheter/robot device techniques and/or structure for characterizing the target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) for use in at least one optical device, assembly, or system to achieve consistent, reliable detection, and/or characterization results at high efficiency and a reasonable cost of manufacture and maintenance.
Accordingly, it is a broad object of the present disclosure to provide imaging (e.g., OCT, NIRF, NIRAF, robots, continuum robots, etc.) apparatuses, systems, methods and storage mediums for using and/or controlling multiple imaging modalities, that apply machine learning, especially deep learning, to evaluate and characterize tissue in one or more images (e.g., intravascular images) with greater or maximum success, and that use the results to achieve tissue characterization more efficiently or with maximum efficiency. It is also a broad object of the present disclosure to provide OCT devices, systems, methods, and storage mediums using an interference optical system, such as an interferometer (e.g., spectral-domain OCT (SD-OCT), swept-source OCT (SS-OCT), multimodal OCT (MM-OCT), Intravascular Ultrasound (IVUS), Near-Infrared Autofluorescence (NIRAF), Near-Infrared Spectroscopy (NIRS), Near-Infrared Fluorescence (NIRF), therapy modality using light, sound, or other source of radiation, etc.).
Further, it is a broad object of the present disclosure to provide one or more methods or techniques that operate to one or more of the following: (i) detect one or more tissue types (e.g., calcium, lipids, fibrous tissue, mixed tissue, etc.) automatically in an entire or whole pullback of catheter or probe for one or more intravascular images (such as, but not limited to, OCT images); (ii) reduce computational time to characterize the pullback by processing one image (e.g., one image only may be processed instead of a plurality (e.g., 400) of images, a carpet view image may be constructed and processed, an intravascular image may be constructed and processed, etc.) in one or more embodiments; (iii) provide one or more methods that do not require any segmentation prior to tissue characterization; and/or (iv) perform a more detailed tissue detection or characterization since spatial connection of tissue in adjacent frames is taken into consideration in one or more embodiments. For example, by constructing one image (e.g., a carpet view) from the pullback, the process allows the tissue characteristics of adjacent frames to be viewed more realistically (as one tissue which is measurable in length) and to be better characterized by a machine learning (ML) algorithm(s) and/or deep learning algorithm(s) (e.g., in a case where a hardware and/or processor setting is already present or included) since each pixel characterization is based on the values of its neighborhood or neighboring pixels too in one or more embodiments.
To overcome the aforementioned issues of manually performing OCT and plaque characterization, several methodologies of the present disclosure have been developed which use either the image (pixel based approach(es)) or the signal (A-line based approach(es)) of the OCT. Using the A-line based approaches, a length of a tissue are may be calculated, and/or using the pixel based approaches, a tissue area also may be quantified. One or more A-line based approaches may include features such as those discussed in J. Lee et al., “Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features,” Sci Rep, vol. 10, no. 1, December 2020, doi: 10.1038/s41598-020-59315-6 and G. van Soest et al., “Atherosclerotic tissue characterization in vivo by optical coherence tomography attenuation imaging,” J Biomed Opt, vol. 15, no. 1, p. 11105, January-February 2010, doi: Artn 011105 Doi 10.1117/1.3280271, the disclosures of which are incorporated by reference herein in their entireties. One or more pixel based approaches may include features such as those discussed in L. S. Athanasiou et al., “Methodology for fully automated segmentation and plaque characterization in intracoronary optical coherence tomography images,” J Biomed Opt, vol. 19, no. 2, p. 026009, February 2014, doi: 10.1117/1.JBO.190.2.026009 and L. S. Athanasiou, M. L. Olender, J. M. de La Torre Hernandez, E. Ben-Assa, and E. R. Edelman, “A deep learning approach to classify atherosclerosis using intracoronary optical coherence tomography,” in Progress in Biomedical Optics and Imaging—Proceedings of SPIE, 2019, vol. 10950. doi: 10.1117/12.2513078, the disclosures of which are incorporated by reference herein in their entireties.
To overcome the aforementioned issues of using segmentation and decision trees, one or more embodiments of the present disclosure may construct an image or images and process the constructed image(s) to detect different tissue types, which minimizes computational time. Additionally, one or more embodiments of the present disclosure may construct the image(s) from the pullback frames (e.g., all of the pullback images), which allows the tissue characterizations of adjacent frames to be viewed more realistically (e.g., as one tissue which is measurable in length) and, therefore, to be better or more efficiently detected by machine learning, deep learning, or other artificial intelligence algorithm(s).
In one or more embodiments, a method for detecting tissue (e.g., calcium, lipids, fibrous tissue, mixed tissue, etc.) in one or more intravascular images (e.g., in one or more OCT images, in one or more carpet view images, in one or more other intravascular images discussed herein, in one or more other intravascular images known to those skilled in the art, etc.) may include one or more of the following features: constructing one image (e.g., a carpet view) while the pullback (e.g., an OCT pullback, an intravascular pullback, etc.) is performed and processing only the one image; detecting all the target tissue (e.g., calcium, lipid, etc.) areas, independent of size, since each tissue (e.g., calcium, lipid, etc.) is analyzed as a connected entity (for example, plaques may be formed and expanded along the artery as an entity and not in individual small segments corresponding to one or more intravascular (e.g., OCT) frames; and constructing and characterizing only one image without the need or use of any pre-required algorithm or performing any other measurements such that the method is free of any error propagation (error which, for example, may be transferred from a pre-tissue detection algorithm to the tissue detection algorithm).
In one or more embodiments of the present disclosure, neural nets may evaluate image(s) faster than any human operator, and neural nets may be deployed across an unlimited number of devices and/or systems. This avoids the issue related to training human operators to evaluate tissue(s) and tissue characteristic(s), and the shorter time required for evaluation reduces the chances of harm to a patient or object, tissue, or specimen by shortening active collection and/or imaging time. Additionally, in one or more embodiments, neural net classifiers may be used to detect specific objects (such as, but not limited to, a catheter sheath, robot components, lipid(s), plaque(s), calcium, calcified areas, etc.) such that more useful information is obtained, evaluated, and used (in comparison with evaluating the tissue(s) or object(s) without a neural net, which provides limited, signal poor, or inaccurate information).
Using artificial intelligence, for example (but not limited to), deep/machine learning, residual learning, a computer vision task (keypoint or object detection and/or image segmentation), using a unique architecture structure of a model or models, using a unique training process, using input data preparation techniques, using input mapping to the model, using post-processing and interpretation of the output data, etc., one or more embodiments of the present disclosure may achieve a better or maximum success rate of tissue or tissue characteristic evaluation(s) without (or with less) user interactions, and may reduce processing and/or prediction time to display tissue or tissue characterization result(s). In the present disclosure, a model may be defined as software that takes images as input and returns predictions for the given images as output. In one or more embodiments, one image may be used as the input to generate accurate, useful output data regarding tissue(s) or tissue characteristic(s). In one or more embodiments a model may be a particular instance of a model architecture (set of parameter values) that has been obtained by model training and selection using a machine (and/or deep) learning and/or optimization algorithm/process. A model may consist or may be comprised of the following parts: an architecture defined by a source code (e.g., a convolutional neural network comprised of layers of parameterized convolution kernels and activation functions, a neural network, a deep neural network(s), a recurrent neural network, etc.) and configuration values (parameters, weights, or features) that are initially set to random values and are then over the course of the training iteratively optimized given data examples (e.g., image-label pairs, tissue data in an image or images, etc.), an objective function (loss function), and an optimization algorithm (optimizer).
Neural networks are a computer system/systems, and take inspiration from how neurons in a brain work. In one or more embodiments, a neural network may consist of or may comprise an input layer, some hidden layers of neurons or nodes, and an output layer. The input layer may be where the values are passed to the rest of the model. In MM-OCT application(s), the input layer may be the place where the transformed OCT data may be passed to a model for evaluation. In one or more embodiments, the hidden layer(s) may be a series of layers that contain or include neurons or nodes that establish connections between the neurons or nodes in the other hidden layers. Through training, the values of each of the connections may be altered so that, due to the training, the system/systems will trigger when the expected pattern is detected. The output layer provides the result(s) of the model. In the case of the MM-OCT application(s), this may be a Boolean (true/false) value for detecting one or more tissues and/or tissue characteristic(s).
In one or more embodiments, a model (which, in one or more embodiments, may be software, software/hardware combination, or a procedure that utilizes one or more machine or deep learning algorithms/procedures/processes that has/have been trained on data to make one or more predictions for future, unseen data) has enough resolution to predict and/or evaluate the tissue(s) and/or characteristic(s) of the tissue(s) with sufficient accuracy depending on the application or procedure being performed. The performance of the model may be further improved by subsequently adding more training data and retraining the model to create a new instance of the model with better or optimized performance. For example, additional training data may include data based on user input, where the user may identify or correct the location of a tissue or tissues and/or a characteristic(s) of the tissue(s) in an image.
In one or more embodiments, an apparatus for detecting and/or characterizing one or more tissues in one or more images may include: one or more processors that operate to: (i) perform a pullback of a catheter or probe and/or obtain one or more images or frames from the pullback of the catheter or probe; (ii) create or construct a Carpet View Image (CVI) based on the one or more images or frames from the pullback or otherwise receive or obtain the CVI; (iii) detect or identify tissue type(s) of one or more tissues shown in the CVI, and/or determine one or more characteristics of the one or more tissues, including whether the one or more tissues is a calcium, a lipid, or another type of tissue; (iv) update the CVI by overlaying information on the CVI to indicate the detected or identified tissue type(s) and/or the determined one or more characteristics of the one or more tissues; and (v) display the updated CVI, or the updated CVI with one or more images or frames from the pullback on a display, or store the updated CVI in a memory. In one or more embodiments, the one or more processors may further operate to one or more of the following: (i) detect one or more tissue types automatically in the pullback of the catheter or the probe for one or more intravascular or Optical Coherence Tomography (OCT) images, where the one or more tissue types include the calcium type, the lipid(s) type, a fibrous tissue type, a mixed tissue type, or the another tissue type; (ii) reduce computational time to characterize the pullback by processing one image only, where the CVI is the one image; (iii) not require any segmentation prior to tissue characterization; (iv) perform a more detailed tissue detection or characterization since a spatial connection or connections of tissue in adjacent frames of the pullback, and/or each pixel characterization is based on values of its neighborhood or neighboring pixels, is taken into consideration and/or since the spatial connection(s) and/or neighboring or neighborhood pixel(s) consideration is better characterized by artificial intelligence, Machine Learning (ML), and/or deep learning networks, structure, or algorithm(s) useable by the one or more processors; (v) use A-line based approaches so that a length of a tissue are is calculated by the one or more processors; and/or (vi) use pixel based approaches so that a tissue area is quantified by the one or more processors. In one or more embodiments, the one or more processors may further operate to perform one or more of the following: display the CVI for further processing and/or display the CVI with one or more intravascular or Optical Coherence Tomography (OCT) images; co-register high texture carpet view areas of the CVI with one or more intravascular or Optical Coherence Tomography (OCT) images; overlay a line on one or more intravascular or Optical Coherence Tomography (OCT) images corresponding to a border indicating the presence of a first type of tissue or the calcium where the line is either a solid line or a dashed or dotted line, and/or overlay a different line on the one or more intravascular or OCT images corresponding to a border indicating the presence of a second type of tissue or the lipid(s) where the different line is the other of the solid line or the dashed or dotted line; and/or display two copies of the CVI, where a first copy of the CVI includes lines overlaid on a first copy of the CVI where each of the lines indicate a border in one or more intravascular or Optical Coherence Tomography (OCT) images and where a second copy of the CVI includes first annotated area(s) corresponding to a first tissue or calcified area(s) and second annotated area(s) corresponding to a second tissue or lipid area(s). In one or more embodiments, the one or more processors may further operate to: in a case where the high texture carpet view areas are detected by the one or more processors, form the high texture carpet view areas due to a presence of sharp edges in the A-line frames or images which represent calcium in the one or more intravascular or OCT images; and in a case where dark homogenous areas are detected by the one or more processors, form or associate corresponding dark homogenous areas to represent lipid(s). The one or more processors may further operate to use AI network(s) and Machine Learning (ML) or other AI-based features to train models to automatically detected calcium and/or lipids based on the use of the high texture carpet view areas representing calcium and based on the dark homogenous areas representing lipid(s). The one or more processors may further operate to use the trained models and/or trained AI network(s) on the CVI only to determine or identify and/or characterize the tissue or tissue characteristics. The one or more processors may further operate to: construct a patch of length L around each pixel of the CVI; extract a set of intensity and texture features from the patch or pixel and classify the pixel to the lipid tissue type, the calcium tissue type, or the another tissue type using artificial intelligence, where the artificial intelligence is one of: Machine Learning (ML), random forests, support vector machines (SVM), and/or another AI-based method, network, or feature; or auto-extract and auto-define a set of intensity and texture features from the patch or pixel using a neural network, convolutional neural network, or other AI-based method or feature and classify the pixel to the lipid tissue type, the calcium tissue type, or the another tissue type; and point, mark, or otherwise indicate, in one or more intravascular or Optical Coherence Tomography (OCT) images, where the calcium and/or lipid starts and ends by using a column part of each corresponding area detected by the one or more processors. The one or more processors may further operate to one or more of the following: (i) set a pixel value of 1; (ii) perform construction of the patch based on the CVI; (iii) perform AI processing by using an AI network or other AI structure to obtain or generate pre-trained classifier(s) or patch feature extraction(s) and to obtain or generate a pre-trained Machine Learning (ML) classifier(s); (iv) identify or characterize whether the pixel being evaluated for the patch construction is a calcium pixel, a lipid pixel, or another type of pixel; (v) perform calcium and lipid pixel translation to a cross sectional intravascular or Optical Coherence Tomography (OCT) image or frame and/or to the one or more intravascular or OCT images; and/or (vi) determine whether the pixel value is less than a value for the number of A-lines x, or by, the number of pullback frames, and, in a case where the pixel value is less than the value for the number of A-lines x, or by, the number of pullback frames, then add 1 to the pixel value and repeat limitations (ii) through (vi) for the next pixel being evaluated, or in a case where the pixel value is greater than or equal to the value for the number of A-lines x, or by, the number of pullback frames, then complete the tissue characterization. The one or more processors may further operate to display results of the tissue characterization completion on the display, store the results in the memory, or use the results to train one or more models or AI-networks to auto-detect or auto-characterize the tissue. The one or more processors may further operate to: indicate a calcium pixel or patch using a solid line and/or indicate a lipid pixel or patch using a dotted or dashed line overlaid on the one or more intravascular or OCT images and/or on the CVI; and/or perform a more detailed tissue detection by taking into consideration a spatial connection or connections of tissue in adjacent frames of the pullback.
In one or more embodiments, the trained model may be one or a combination of the following: a neural net model or neural network model, a deep convolutional neural network model, a recurrent neural network model with long short-term memory that can take temporal relationships across images or frames into account, a generative adversarial network (GAN) model, a consistent generative adversarial network (cGAN) model, a three cycle-consistent generative adversarial network (3cGAN) model, a model that can take temporal relationships across images or frames into account, a model that can take temporal relationships into account including tissue location(s) during pullback in a vessel and/or including tissue characterization data during pullback in a vessel, a model that can use prior knowledge about a procedure and incorporate the prior knowledge into the machine learning algorithm or a loss function, a model using feature pyramid(s) that can take different image resolutions into account, and/or a model using residual learning technique(s); a segmentation model, a segmentation model with post-processing, a model with pre-processing, a model with post-processing, a segmentation model with pre-processing, a deep learning or machine learning model, a semantic segmentation model or classification model, an object detection or regression model, an object detection or regression model with pre-processing or post-processing, a combination of a semantic segmentation model and an object detection or regression model, a model using repeated segmentation model technique(s), a model using feature pyramid(s), a genetic algorithm that operates to breed multiple models for improved performance, and/or a model using repeated object detection or regression model technique(s).
In one or more embodiments, an apparatus may have one or more of the following occur: the CVI has dimensions equal to a number of A-lines (A) of each A-line frame times, or by, a number of pullback cross sectional frames or images or by a number of total frames (N), and setting a counter i to a value of 1; and the one or more processors further operate to: (i) determine whether i is less than or equal to the number of pullback frames, N; and (ii) in a case where i is less than or equal to N and until i is more than N, repeat the performance of the following: acquire an A-line frame corresponding to the value for i; perform or apply thresholding or automatic thresholding for the acquired A-line frame or image; summarize each line of the thresholded image or frame, and/or compress the thresholded image or frame in one dimension (1D) by summing columns of the thresholded image or frame so that each line of the thresholded image or frame is in 1D; and add the 1D line to the ith line of the created or constructed CVI, and add 1 to the value for i; and/or (iii) in a case where i is more than N such that all of the pullback frames, N, have been processed, show, reveal, display on a display, and/or store the created or constructed CVI in the memory. The constructed or created CVI may be saved in the memory and/or may be sent to the one or more processors or an artificial intelligence (AI) network for use in AI evaluations or determinations. The one or more processors may further operate to use one or more neural networks or convolutional neural networks to one or more of: load a trained model of CVI images including calcium and or lipid area(s), create or construct the CVI, evaluate whether the counter i is less than or equal to, or greater than, the number of pullback frames N, detect or identify the tissue type(s) in the CVI, apply the thresholding or automatic thresholding, perform the summarizing or compressing of the thresholded image in 1D by summing the columns of the image, perform the addition of the 1D line to the ith line of the CVI, determine whether the detected or identified tissue type(s) is/are accurate or correct, determine the one or more of the characteristics of the tissue(s), identify or detect the one or more tissues, overlay data on the CVI to show the location(s) of the intravascular image(s) and/or to show the areas for the tissue type(s), display the results for the tissue identification/detection or characterization on a display, and/or acquire or receive the image data during the pullback operation of the catheter or the probe. The one or more processors may further operate to use one or more neural networks or convolutional neural networks to one or more of: incorporate image processing and machine learning (ML) or deep learning to automatically identify and locate calcium and lipid(s); create or construct the CVI for the whole pullback and apply ML only to the CVI; create or construct the CVI or another image having dimensions equal to the number of A-lines (A) of each A-line frame, by the number of total frames (N); acquire the first A-line frame and continue to acquire the A-lines of the frame, threshold the image, summarize each A-line of the thresholded image to generate a one-dimensional (1D) signal or line having size A, add or copy the 1D signal or line in or to a first column of the A×N image, and repeat the acquire, threshold, summarize, and add or copy features for all of the pullback frames so that a next 1D signal or line is added or copied in or to the corresponding next column of the A×N image until all subsequent 1D signals or lines are added or copied in or to the corresponding subsequent, respective columns of the A×N image; and reveal, show, or display the CVI or the A×N image, and/or store the created or constructed CVI in the memory, after the last 1D signal or line is copied or added in or to the last column of the A×N image.
In one or more embodiments, an apparatus may include one or more of the following: a light source that operates to produce a light; an interference optical system that operates to: (i) receive and divide the light from the light source into a first light with which an object or sample is to be irradiated and a second reference light, (ii) send the second reference light for reflection off of a reference mirror of the interference optical system, and (iii) generate interference light by causing reflected or scattered light of the first light with which the object or sample has been irradiated and the reflected second reference light to combine or recombine, and to interfere, with each other, the interference light generating one or more interference patterns; and/or one or more detectors that operate to continuously acquire the interference light and/or the one or more interference patterns such that the one or more lumen edges, the one or more stents, and/or the one or more artifacts are detected in the images, and the one or more stents and/or the one or more artifacts are removed from the one or more images.
In one or more embodiments, a method for detecting and/or characterizing one or more tissues in one or more images may include: (i) performing a pullback of a catheter or probe and/or obtaining one or more images or frames from the pullback of the catheter or probe; (ii) creating or constructing a Carpet View Image (CVI) based on the one or more images or frames from the pullback or otherwise receive or obtain the CVI; (iii) detecting or identifying tissue type(s) of one or more tissues shown in the CVI, and/or determining one or more characteristics of the one or more tissues, including whether the one or more tissues is a calcium, a lipid, or another type of tissue; (iv) updating the CVI by overlaying information on the CVI to indicate the detected or identified tissue type(s) and/or the determined one or more characteristics of the one or more tissues; and (v) displaying the updated CVI, or the updated CVI with one or more images or frames from the pullback on a display, and/or storing the updated CVI in a memory. One or more embodiments of a method of the present disclosure may include one or more features of the apparatuses, storage mediums, artificial intelligence structure or algorithms, or other features discussed herein.
In one or more embodiments, a computer-readable storage medium may store at least one program that operates to cause one or more processors to execute a method for detecting and/or characterizing one or more tissues in one or more images, where the method may include: (i) performing a pullback of a catheter or probe and/or obtaining one or more images or frames from the pullback of the catheter or probe; (ii) creating or constructing a Carpet View Image (CVI) based on the one or more images or frames from the pullback or otherwise receive or obtain the CVI; (iii) detecting or identifying tissue type(s) of one or more tissues shown in the CVI, and/or determining one or more characteristics of the one or more tissues, including whether the one or more tissues is a calcium, a lipid, or another type of tissue; (iv) updating the CVI by overlaying information on the CVI to indicate the detected or identified tissue type(s) and/or the determined one or more characteristics of the one or more tissues; and (v) displaying the updated CVI, or the updated CVI with one or more images or frames from the pullback on a display, and/or storing the updated CVI in a memory. One or more embodiments of a storage medium of the present disclosure may include one or more features of the apparatuses, methods, artificial intelligence structure or algorithms, or other features discussed herein.
While one or more embodiments may use one image, it may be advantageous from a safety perspective to have the neural net/network evaluate more than one frame/image to establish the status of a tissue or tissues or a characteristic(s) of the tissue(s) with more certainty/accuracy. In one or more embodiments, the tissue(s) may be evaluated using artificial intelligence structure, such as, but not limited, convolutional neural networks, generative adversarial networks (GANs), neural networks, any other AI structure or feature(s) discussed herein, any other AI network structure(s) known to those skilled in the art, etc. For example, a generator of a generative adversarial network may operate to generate an image(s) that is/are so similar to ground truth image(s) that a discriminator of the generative adversarial network is not able to distinguish between the generated image(s) and the ground truth image(s). The generative adversarial network may include one or more generators and one or more discriminators. Each generator of the generative adversarial network may operate to estimate tissue(s) or tissue characteristic(s) of each image (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.), and each discriminator of the generative adversarial network may operate to determine whether the estimated tissue(s) or tissue characteristic(s) of each image (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.) is estimated (or fake) or ground truth (or real). In one or more embodiments, an AI network, such as, but not limited to, a GAN or a consistent GAN (cGAN), may receive an image or images as an input and may obtain or create a tissue map (e.g., annotated area(s) representing one or more tissue types or characteristics) for each image or images. In one or more embodiments, an AI network may evaluate obtained one or more images (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.), one or more virtual images, and one or more ground truth tissue maps to generate tissue map(s) for the one or more images and/or evaluate the generated tissue map(s). A Three Cycle-Consistent Generative Adversarial Network (3cGAN) may be used to obtain or construct the tissue map(s), evaluate the quality of the tissue map(s).
In one or more embodiments, an artificial intelligence training apparatus using a neural network or other AI-ready network may include: a memory; one or more processors in communication with the memory, the one or more processors operating to: training a classifier or patch feature extraction and training an AI-classifier (e.g., a ML classifier, a DL classifier, etc.). In one or more embodiments of the present disclosure, an apparatus, a system, or a storage medium may use an AI network, a neural network, or other AI-ready network to perform any of the method step(s) or feature(s) discussed herein.
The one or more processors may further operate to use one or more neural networks, convolutional neural networks, recurrent neural networks, a generative adversarial network (GAN), a consistent generative adversarial network (cGAN), a three cycle-consistent generative adversarial network (3cGAN), and/or any other AI architecture discussed herein or known to those skilled in the art to one or more of: load the trained model, select a set of angiography frames or other type of image frames or select one or more intravascular images, identify one or more targets, objects, or samples (or one or more tissues in the targets, objects, or samples), evaluate the tissue(s) and/or tissue characteristic(s), determine whether the tissue(s) or tissue characteristic(s) evaluation is appropriate with respect to given prior knowledge (for example, vessel location and pullback direction, prior tissue location or characteristic information, etc.), modify the evaluation results or the location of the tissue(s) or characteristic(s) of the tissue(s) for each frame, construct an image based on the tissue(s) or characteristic(s) of the tissue(s), perform coregistration of the constructed image with one or more intravascular images (e.g., OCT images, two-dimensional (2D) OCT images, etc.), insert or overlay intravascular image data (e.g., indicators or lines used to represent one or more tissue types, indicators or lines used to represent calcified (e.g., with a solid line) areas, indicators or lines used to represent lipid (e.g., with a dashed or dotted line)) over the constructed image (e.g., a line may represent a border in each of the intravascular images, a line or an outlined area may represent areas corresponding to different tissue(s) (e.g., calcified areas, lipid areas, etc.)), and/or acquire or receive the image data during the pullback operation.
In one or more embodiments, the object, target, or sample may include one or more of the following: a vessel, a target specimen or object, a tissue or tissues, and a patient.
The one or more processors may further operate to perform the coregistration by co-registering an acquired or received angiography image or the constructed image (e.g., a carpet view) and an obtained one or more intravascular images, such as, but not limited to, OCT or IVUS images or frames.
In one or more embodiments, a loaded, trained model may be one or a combination of the following: a segmentation (classification) model, a segmentation model with pre-processing, a segmentation model with post-processing, an object detection (regression) model, an object detection model with pre-processing, an object detection model with post-processing, a combination of a segmentation (classification) model and an object detection (regression) model, a deep convolutional neural network model, a recurrent neural network model with long short-term memory that can take temporal relationships across images or frames into account, a model using feature pyramid(s) that can take different image resolutions into account, a genetic algorithm that operates to breed multiple models for improved performance (as compared with a case where the genetic algorithm is not used), a model using residual learning technique(s), and/or any other model discussed herein or known to those skilled in the art.
In one or more embodiments, the one or more processors may further operate to one or more of the following: (i) display an image for each of one or more imaging modalities on a display, wherein the one or more imaging modalities include one or more of the following: a tomography image; an Optical Coherence Tomography (OCT) image; a fluorescence image; a near-infrared auto-fluorescence (NIRAF) image; a near-infrared auto-fluorescence (NIRAF) image in a predetermined view, a carpet view, and/or an indicator view; a near-infrared fluorescence (NIRF) image, a near-infrared fluorescence (NIRF) image in a predetermined view, a carpet view, and/or an indicator view; a three-dimensional (3D) rendering; a 3D rendering of a vessel; a 3D rendering of a vessel in a half-pipe view or display; a 3D rendering of the object; a lumen profile; a lumen diameter display; a longitudinal view; computer tomography (CT); Magnetic Resonance Imaging (MRI); Intravascular Ultrasound (IVUS); an X-ray image or view; and an angiography view; and (ii) change or update the displays based on the tissue(s) or tissue characteristic(s) evaluation results and/or an updated location of the catheter.
One or more embodiments of a non-transitory computer-readable storage medium storing at least one program for causing a computer to execute a method for training a model using artificial intelligence may be used with any method(s) discussed in the present disclosure, including but not limited to, one or more tissue(s) or tissue characteristic(s) evaluation/determination method(s).
One or more embodiments of any method discussed herein (e.g., training method(s), detecting method(s), imaging or visualization method(s), artificial intelligence method(s), etc.) may be used with any feature or features of the apparatuses, systems, other methods, storage mediums, or other structures discussed herein.
One or more of the artificial intelligence features discussed herein that may be used in one or more embodiments of the present disclosure, includes but is not limited to, using one or more of deep learning, a computer vision task, keypoint detection, a unique architecture of a model or models, a unique training process or algorithm, a unique optimization process or algorithm, input data preparation techniques, input mapping to the model, pre-processing, post-processing, and/or interpretation of the output data as substantially described herein or as shown in any one of the accompanying drawings.
In one or more embodiments, tissue(s) and or characteristic(s) of one or more tissues may be evaluated and determined using an algorithm, such as, but not limited to, the Viterbi algorithm.
One or more embodiments of the present disclosure may track and/or calculate a tissue(s) or tissue characteristic(s) evaluation success rate.
The following paragraphs describe certain explanatory embodiments. Other embodiments may include alternatives, equivalents, and modifications. Additionally, the explanatory embodiments may include several novel features, and a particular feature may not be essential to some embodiments of the devices, systems, and methods that are described herein.
According to other aspects of the present disclosure, one or more additional devices, one or more systems, one or more methods and one or more storage mediums using OCT and/or other imaging modality technique(s) to perform tissue characterization and to perform coregistration using artificial intelligence, including, but not limited to, deep or machine learning, using results of the tissue detection and/or tissue characterization for performing coregistration, etc., are discussed herein. Further features of the present disclosure will in part be understandable and will in part be apparent from the following description and with reference to the attached drawings.
In accordance with one or more embodiments of the present disclosure, apparatuses and systems, and methods and storage mediums for tissue detection and/or tissue characterization in one or more images may operate to characterize biological objects, such as, but not limited to, blood, mucus, tissue (including different types of tissue), etc.
It should be noted that one or more embodiments of the tissue detection and/or characterization method(s) or feature(s) of the present disclosure may be used in other imaging systems, apparatuses or devices, where images are formed from signal reflection and scattering within tissue sample(s) using a scanning probe. For example, IVUS images may be processed in addition to or instead of OCT images.
One or more embodiments of the present disclosure may be used in clinical application(s), such as, but not limited to, intervascular imaging, atherosclerotic plaque assessment, cardiac stent evaluation, balloon sinuplasty, sinus stenting, arthroscopy, ophthalmology, ear research, veterinary use and research, etc.
In accordance with at least another aspect of the present disclosure, one or more technique(s) discussed herein may be employed to reduce the cost of at least one of manufacture and maintenance of the one or more apparatuses, devices, systems and storage mediums by reducing or minimizing a number of optical components and by virtue of the efficient techniques to cut down cost of use/manufacture of such apparatuses, devices, systems and storage mediums.
According to other aspects of the present disclosure, one or more additional devices, one or more systems, one or more methods and one or more storage mediums using, or for use with, one or more tissue detection and/or tissue characterization techniques are discussed herein. Further features of the present disclosure will in part be understandable and will in part be apparent from the following description and with reference to the attached drawings.
For the purposes of illustrating various aspects of the disclosure, wherein like numerals indicate like elements, there are shown in the drawings simplified forms that may be employed, it being understood, however, that the disclosure is not limited by or to the precise arrangements and instrumentalities shown. To assist those of ordinary skill in the relevant art in making and using the subject matter hereof, reference is made to the appended drawings and figures, wherein:
One or more devices, systems, methods and storage mediums for characterizing tissue, or an object, using one or more imaging techniques or modalities (such as, but not limited to, OCT, fluorescence, IVUS, MRI, CT, NIRF, NIRAF, etc.), and using artificial intelligence for evaluating and detecting tissue types and/or characteristics and/or performing coregistration are disclosed herein. Several embodiments of the present disclosure, which may be carried out by the one or more embodiments of an apparatus, system, method and/or computer-readable storage medium of the present disclosure are described diagrammatically and visually in at least
One or more embodiments of the present disclosure provide at least one imaging or optical apparatus/device, system, method, and storage medium that applies machine learning and/or deep learning to evaluate and characterize a target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) with a higher success rate when compared to traditional techniques (for example, techniques performed manually by an operator or user), and to use the one or more results to characterize the target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) more efficiently. One or more embodiments of the present disclosure may also provide or use one or more probe/catheter/robot device techniques and/or structure for characterizing the target, sample, or object (e.g., a tissue, an organ, a part of a patient, a vessel, etc.) for use in at least one optical device, assembly, or system to achieve consistent, reliable detection, and/or characterization results at high efficiency and a reasonable cost of manufacture and maintenance.
One or more embodiments of the present disclosure provide imaging (e.g., OCT, NIRF, NIRAF, robots, continuum robots, etc.) apparatuses, systems, methods and storage mediums for using and/or controlling multiple imaging modalities, that apply machine learning, especially deep learning, to evaluate and characterize tissue in one or more images (e.g., intravascular images) with greater or maximum success, and that use the results to achieve tissue characterization more efficiently or with maximum efficiency. One or more embodiments of the present disclosure may operate to provide OCT devices, systems, methods, and storage mediums using an interference optical system, such as an interferometer (e.g., spectral-domain OCT (SD-OCT), swept-source OCT (SS-OCT), multimodal OCT (MM-OCT), Intravascular Ultrasound (IVUS), Near-Infrared Autofluorescence (NIRAF), Near-Infrared Spectroscopy (NIRS), Near-Infrared Fluorescence (NIRF), therapy modality using light, sound, or other source of radiation, etc.).
Further, one or more embodiments of the present disclosure may provide one or more methods or techniques that operate to one or more of the following: (i) detect one or more tissue types (e.g., calcium, lipids, fibrous tissue, mixed tissue, etc.) automatically in an entire or whole pullback of catheter or probe for one or more intravascular images (such as, but not limited to, OCT images); (ii) reduce computational time to characterize the pullback by processing one image (e.g., one image only may be processed instead of a plurality (e.g., 400) of images, a carpet view image may be constructed and processed, an intravascular image may be constructed and processed, etc.) in one or more embodiments; (iii) provide one or more methods that do not require any segmentation prior to tissue characterization; and/or (iv) perform a more detailed tissue detection or characterization since spatial connection of tissue in adjacent frames is taken into consideration in one or more embodiments. For example, by constructing one image (e.g., a carpet view) from the pullback, the process allows the tissue characteristics of adjacent frames to be viewed more realistically (as one tissue which is measurable in length) and to be better characterized by a machine learning (ML) algorithm(s) and/or deep learning algorithm(s) (e.g., in a case where a hardware and/or processor setting is already present or included) since each pixel characterization is based on the values of its neighborhood or neighboring pixels too in one or more embodiments.
To overcome the aforementioned issues of manually performing OCT and plaque characterization, several methodologies of the present disclosure have been developed which use either the image (pixel based approach(es)) or the signal (A-line based approach(es)) of the OCT. Using the A-line based approaches, a length of a tissue are may be calculated, and/or using the pixel based approaches, a tissue area also may be quantified. One or more A-line based approaches may include features such as those discussed in J. Lee et al., “Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features,” Sci Rep, vol. 10, no. 1, December 2020, doi: 10.1038/s41598-020-59315-6 and G. van Soest et al., “Atherosclerotic tissue characterization in vivo by optical coherence tomography attenuation imaging,” J Biomed Opt, vol. 15, no. 1, p. 11105, January-February 2010, doi: Artn 011105 Doi 10.1117/1.3280271, the disclosures of which are incorporated by reference herein in their entireties. One or more pixel based approaches may include features such as those discussed in L. S. Athanasiou et al., “Methodology for fully automated segmentation and plaque characterization in intracoronary optical coherence tomography images,” J Biomed Opt, vol. 19, no. 2, p. 026009, February 2014, doi: 10.1117/1.JBO.190.2.026009 and L. S. Athanasiou, M. L. Olender, J. M. de La Torre Hernandez, E. Ben-Assa, and E. R. Edelman, “A deep learning approach to classify atherosclerosis using intracoronary optical coherence tomography,” in Progress in Biomedical Optics and Imaging—Proceedings of SPIE, 2019, vol. 10950. doi: 10.1117/12.2513078, the disclosures of which are incorporated by reference herein in their entireties.
To overcome the aforementioned issues of using segmentation and decision trees, one or more embodiments of the present disclosure may construct an image or images and process the constructed image(s) to detect different tissue types, which minimizes computational time. Additionally, one or more embodiments of the present disclosure may construct the image(s) from the pullback frames (e.g., all of the pullback images), which allows the tissue characterizations of adjacent frames to be viewed more realistically (e.g., as one tissue which is measurable in length) and, therefore, to be better or more efficiently detected by machine learning, deep learning, or other artificial intelligence algorithm(s). While not limited thereto, one or more embodiments of the present disclosure may construct and use only one image or frame to detect one or more tissue types and/or characteristics, which also achieves reduced processing power and time involved such that the embodiments of the present disclosure achieve greater efficiency (compared to methods that may require construction and evaluation of numerous images to evaluate tissue). One or more other embodiments may select or construct an image from less than all of the pullback images for improved efficiency as well in a case where more than one image would be useful for other benefits (e.g., evaluation result(s) confirmation, patient safety, etc.).
In one or more embodiments, a method for detecting tissue (e.g., calcium, lipids, fibrous tissue, mixed tissue, etc.) in one or more intravascular images (e.g., in one or more OCT images, in one or more carpet view images, in one or more other intravascular images discussed herein, in one or more other intravascular images known to those skilled in the art, etc.) may include one or more of the following features: selecting or constructing one image (e.g., a carpet view, one image including two carpet views, only one image, only one carpet view, only one image including two carpet views, selecting an image from a plurality of images, etc.) while the pullback (e.g., an OCT pullback, an intravascular pullback, etc.) is performed and processing only the one image; detecting all the target tissue (e.g., calcium, lipid, etc.) areas, independent of size, since each tissue (e.g., calcium, lipid, etc.) is analyzed as a connected entity (for example, plaques may be formed and expanded along the artery as an entity and not in individual small segments corresponding to one or more intravascular (e.g., OCT) frames; and constructing and characterizing only one image without the need or use of any pre-required algorithm or performing any other measurements such that the method is free of any error propagation (error which, for example, may be transferred from a pre-tissue detection algorithm to the tissue detection algorithm).
In one or more embodiments of the present disclosure, neural nets may evaluate image(s) faster than any human operator, and neural nets may be deployed across an unlimited number of devices and/or systems. This avoids the issue related to training human operators to evaluate tissue(s) and tissue characteristic(s), and the shorter time required for evaluation reduces the chances of harm to a patient or object, tissue, or specimen by shortening active collection and/or imaging time. Additionally, in one or more embodiments, neural net classifiers may be used to detect specific objects (such as, but not limited to, a catheter sheath, robot components, lipid(s), plaque(s), calcium, calcified areas, etc.) such that more useful information is obtained, evaluated, and used (in comparison with evaluating the tissue(s) or object(s) without a neural net, which provides limited, signal poor, or inaccurate information).
Using artificial intelligence, for example (but not limited to), deep/machine learning, residual learning, a computer vision task (keypoint or object detection and/or image segmentation), using a unique architecture structure of a model or models, using a unique training process, using input data preparation techniques, using input mapping to the model, using post-processing and interpretation of the output data, etc., one or more embodiments of the present disclosure may achieve a better or maximum success rate of tissue or tissue characteristic evaluation(s) without (or with less) user interactions, and may reduce processing and/or prediction time to display tissue or tissue characterization result(s). In the present disclosure, a model may be defined as software that takes images as input and returns predictions for the given images as output. In one or more embodiments, one image may be used as the input to generate accurate, useful output data regarding tissue(s) or tissue characteristic(s). In one or more embodiments a model may be a particular instance of a model architecture (set of parameter values) that has been obtained by model training and selection using a machine (and/or deep) learning and/or optimization algorithm/process. A model may consist or may be comprised of the following parts: an architecture defined by a source code (e.g., a convolutional neural network comprised of layers of parameterized convolution kernels and activation functions, a neural network, a deep neural network(s), a recurrent neural network, etc.) and configuration values (parameters, weights, or features) that are initially set to random values and are then over the course of the training iteratively optimized given data examples (e.g., image-label pairs, tissue data in an image or images, etc.), an objective function (loss function), and an optimization algorithm (optimizer).
Neural networks are a computer system/systems, and take inspiration from how neurons in a brain work. In one or more embodiments, a neural network may consist of or may comprise an input layer, some hidden layers of neurons or nodes, and an output layer. The input layer may be where the values are passed to the rest of the model. In MM-OCT application(s), the input layer may be the place where the transformed OCT data may be passed to a model for evaluation. In one or more embodiments, the hidden layer(s) may be a series of layers that contain or include neurons or nodes that establish connections between the neurons or nodes in the other hidden layers. Through training, the values of each of the connections may be altered so that, due to the training, the system/systems will trigger when the expected pattern is detected. The output layer provides the result(s) of the model. In the case of the MM-OCT application(s), this may be a Boolean (true/false) value for detecting one or more tissues and/or tissue characteristic(s).
In one or more embodiments, a model (which, in one or more embodiments, may be software, software/hardware combination, or a procedure that utilizes one or more machine or deep learning algorithms/procedures/processes that has/have been trained on data to make one or more predictions for future, unseen data) has enough resolution to predict and/or evaluate the tissue(s) and/or characteristic(s) of the tissue(s) with sufficient accuracy depending on the application or procedure being performed. The performance of the model may be further improved by subsequently adding more training data and retraining the model to create a new instance of the model with better or optimized performance. For example, additional training data may include data based on user input, where the user may identify or correct the location of a tissue or tissues and/or a characteristic(s) of the tissue(s) in an image.
While one or more embodiments may use one image, it may be advantageous from a safety perspective to have the neural net/network evaluate more than one frame/image to establish the status of a tissue or tissues or a characteristic(s) of the tissue(s) with more certainty/accuracy. In one or more embodiments, the tissue(s) may be evaluated using artificial intelligence structure, such as, but not limited, convolutional neural networks, generative adversarial networks (GANs), neural networks, any other AI structure or feature(s) discussed herein, any other AI network structure(s) known to those skilled in the art, etc. For example, a generator of a generative adversarial network may operate to generate an image(s) that is/are so similar to ground truth image(s) that a discriminator of the generative adversarial network is not able to distinguish between the generated image(s) and the ground truth image(s). The generative adversarial network may include one or more generators and one or more discriminators. Each generator of the generative adversarial network may operate to estimate tissue(s) or tissue characteristic(s) of each image (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.), and each discriminator of the generative adversarial network may operate to determine whether the estimated tissue(s) or tissue characteristic(s) of each image (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.) is estimated (or fake) or ground truth (or real). In one or more embodiments, an AI network, such as, but not limited to, a GAN or a consistent GAN (cGAN), may receive an image or images as an input and may obtain or create a tissue map (e.g., annotated area(s) representing one or more tissue types or characteristics) for each image or images. In one or more embodiments, an AI network may evaluate obtained one or more images (e.g., a CT image, an OCT image, an IVUS image, a bronchoscopic image, etc.), one or more virtual images, and one or more ground truth tissue maps to generate tissue map(s) for the one or more images and/or evaluate the generated tissue map(s). A Three Cycle-Consistent Generative Adversarial Network (3cGAN) may be used to obtain or construct the tissue map(s), evaluate the quality of the tissue map(s).
Turning now to the details of the figures, imaging modalities may be displayed in one or more ways as discussed herein. One or more displays discussed herein may allow a user of the one or more displays to use, control and/or emphasize multiple imaging techniques or modalities, such as, but not limited to, OCT, CT, IVUS, NIRF, NIRAF, etc., and may allow the user to use, control, and/or emphasize the multiple imaging techniques or modalities synchronously.
As shown diagrammatically in
In medical procedures, improvement or optimization of physiological assessment is preferable to decide a course of treatment for a particular patient. By way of at least one example, physiological assessment is very useful for deciding treatment for cardiovascular disease patients. In a catheterization lab, for example, physiological assessment may be used as a decision-making tool—e.g., whether a patient should undergo a PCI procedure, whether a PCI procedure is successful, etc. While the concept of using physiological assessment is theoretically sound, physiological assessment still waits for more adaption and improvement for use in the clinical setting(s). This situation may be because physiological assessment may involve adding another device and medication to be prepared, and/or because a measurement result may vary between physicians due to technical difficulties. Such approaches add complexities and lack consistency. Therefore, one or more embodiments of the present disclosure may employ computational fluid dynamics based (CFD-based) physiological assessment that may be performed from imaging data to eliminate or minimize technical difficulties, complexities and inconsistencies during the measurement procedure. To obtain accurate physiological assessment, an accurate 3D structure of the vessel may be reconstructed from the imaging data as disclosed in U.S. Provisional Pat. App. No. 62/901,472, filed on Sep. 17, 2019, the disclosure of which is incorporated by reference herein in its entirety, and as disclosed in U.S. patent application Ser. No. 16/990,800, filed Aug. 11, 2020, the disclosure of which is incorporated by reference herein in its entirety. Additionally or alternatively, the determination or identification of one or more tissue types and/or tissue characteristics operates to provide additional information for physiological assessment.
In at least one embodiment of the present disclosure, a method may be used to provide more accurate 3D structure(s) compared to using only one imaging modality. In one or more embodiments, a combination of multiple imaging modalities may be used, one or more tissue types and/or tissue characteristics may be detected, and coregistration may be processed/performed using artificial intelligence.
One or more embodiments of the present disclosure may apply machine learning, especially deep learning, to detect one or more tissue types and/or tissue characteristics in an image frame without user input(s) that define an area where intravascular imaging pullback occurs. Using artificial intelligence, for example, deep learning, one or more embodiments of the present disclosure may achieve a better or maximum success rate of tissue type(s) and/or tissue characteristic(s) detection from image data without (or with less) user interactions, and may reduce processing and/or prediction time to display coregistration result(s) based on the tissue type(s) and/or tissue characteristic(s) detection result(s) and/or based on the improved image quality obtained when detecting tissue type(s) and/or tissue characteristic(s).
One or more embodiments of the present disclosure may achieve the efficient catheter (or other imaging device) detection of tissue type(s) and/or tissue characteristic(s) and/or efficient coregistration result(s) from image(s). In one or more embodiments, the image data may be acquired during intravascular imaging pullback using a catheter (or other imaging device) that may be visualized in an image. In one or more embodiments, a ground truth identifies a location or locations of the catheter or a portion of the catheter (or of another imaging device or a portion of the another imaging device). In one or more embodiments, a model has enough resolution to predict the tissue type(s) and/or tissue characteristic(s) (e.g., location, size, etc.) in a given image with sufficient accuracy depending on the application or procedure being performed. The performance of the model may be further improved by adding more training data. For example, additional training data may include image annotations, where a user labels or corrects the tissue type(s), tissue characterization(s), and/or catheter detection(s) in each image.
In one or more embodiments, one or more tissue type(s) and/or tissue characteristic(s) may be detected and/or monitored using an algorithm, such as, but not limited to, the Viterbi algorithm.
One or more embodiments may automate characterization of tissue(s) and/or identification of tissue type(s) in images using convolutional neural networks (or other AI structure discussed herein or known to those skilled in the art), and may fully automate frame detection on angiographies, intravascular pullbacks, etc. using training (e.g., offline training) and using applications (e.g., online application(s)) to extract and process frames via deep learning.
One or more embodiments of the present disclosure may track and/or calculate a tissue type(s) and/or tissue characteristic(s) detection or identification success rate.
In at least one further embodiment example, a method of 3D reconstruction without adding any imaging requirements or conditions may be employed. One or more methods of the present disclosure may use intravascular imaging, e.g., IVUS, OCT, etc., and one (1) view of angiography. One or more embodiments may use one image only (e.g., carpet view, a frame of carpet views, another frame or image type discussed herein or known to those skilled in the art, etc.) In the description below, while intravascular imaging of the present disclosure is not limited to OCT, OCT is used as a representative of intravascular imaging for describing one or more features herein.
Referring now to
The intravascular imaging system 40 of the imaging system 20 may include a console 32, a catheter 120, and a patient interface unit or PIU 110 that connects between the catheter 120 and the console 32 for acquiring intravascular image frames. The catheter 120 may be inserted into a blood vessel of the patient 1o6 (or inside a specimen or other target object, inside tissue, etc.). The catheter 120 may function as a light irradiator and a data collection probe that is disposed in a lumen of a particular blood vessel, such as, for example, a coronary artery, or in another type of tissue or specimen. The catheter 120 may include a probe tip, one or more markers or radiopaque markers, an optical fiber, and a torque wire. The probe tip may include one or more data collection systems. The catheter 120 may be threaded in an artery of the patient 106 to obtain images of the coronary artery. The patient interface unit 110 may include a motor M inside to enable pullback of imaging optics during the acquisition of intravascular image frames. The imaging pullback procedure may obtain images of the blood vessel. The imaging pullback path may represent the co-registration path, which may be a region of interest or a targeted region of the vessel.
The console 32 may include a light source(s) 101 and a computer 1200. The computer 1200 may include features as discussed herein and below (see e.g.,
Various types of intravascular imaging systems may be used within the imaging system 20. The intravascular imaging system 40 is merely one example of an intravascular imaging system that may be used within the imaging system 20. Various types of intravascular imaging systems may be used, including, but not limited to, an OCT system, a multi-modality OCT system or an IVUS system, by way of example.
The imaging system 20 may also connect to an electrocardiography (ECG) device (or other monitoring device) 60 for recording the electrical activity of the heart (or other organ being monitored, tissue being monitored, specimen being monitored, etc.) over a period of time using electrodes placed on the skin of the patient 106. The imaging system 20 may also include an image processor 50 for receiving angiography data, intravascular imaging data, and data from the ECG device 60 to execute various image-processing steps to transmit to a display 1209 for displaying an angiography image frame with a co-registration path. Although the image processor 50 associated with the imaging system 20 appears external to both the angiography system 20 and the intravascular imaging system 30 in
To collect data that may be used to train one or more neural nets, one or more features of an OCT device or system (e.g., an MM-OCT device or system, a SS-OCT device or system, etc.) may be used. Collecting a series of OCT images with or without tissue(s) being shown, with one or more tissue types, with one or more tissue characteristics, etc. may result in a plurality (e.g., several thousand) of training images. In one or more embodiments, the data may be labeled based on whether a tissue type was identified or detected, a tissue characteristic was identified or detected, a tissue location was identified or detected, a plurality of tissue types are detected, a plurality of tissue characteristics are detected, etc. (as confirmed by a trained operator or user of the device or system). In one or more embodiments, after at least 30,000 OCT images are captured and labeled, the data may be split into a training population and a test population. In one or more embodiments, data collection may be performed in the same environment or in different environments. For example, during data collection, a flashlight (or any light source) may be used to shine the light down a barrel of an imaging device with no catheter imaging core to confirm that a false positive would not occur in a case where a physician pointed the imaging device at external lights (e.g., operating room lights, a computer screen, etc.). After training is complete, the testing data may be fed through the neural net or neural networks, and the accuracy of the model(s) may be evaluated based on the result(s) of the test data.
The catheter 120, which, in one or more embodiments, comprises the sheath 121, the coil 122, the protector 123 and the optical probe 124 as aforementioned (and as shown in
As aforementioned, in one or more embodiments, the coil 122 delivers torque from a proximal end to a distal end thereof (e.g., via or by a rotational motor in the PIU 110). There may be a mirror at the distal end so that the light beam is deflected outward. In one or more embodiments, the coil 122 is fixed with/to the optical probe 124 so that a distal tip of the optical probe 124 also spins to see an omnidirectional view of an object (e.g., a biological organ, sample or material being evaluated, such as, but not limited to, hollow organs such as vessels, a heart, a coronary artery, etc.). In one or more embodiments, the optical probe 124 may include a fiber connector at a proximal end, a double clad fiber and a lens at distal end. The fiber connector operates to be connected with the PIU 11o. The double clad fiber may operate to transmit & collect OCT light through the core and, in one or more embodiments, to collect Raman and/or fluorescence from an object (e.g., the object 106 (e.g., a vessel) discussed herein, an object and/or a patient (e.g., a vessel in the patient), etc.) through the clad. The lens may be used for focusing and collecting light to and/or from the object (e.g., the object 1o6 (e.g., a vessel) discussed herein). In one or more embodiments, the scattered light through the clad is relatively higher than that through the core because the size of the core is much smaller than the size of the clad.
Embodiments of a method or methods for detecting or identifying one or more tissue types and/or tissue characteristics may be used independently or in combination. While not limited to the discussed combination or arrangement, one or more steps may be involved in both of the workflows or processes in one or more embodiments of the present disclosure, for example, as shown in one or more of
Once trained, a neural network or networks may use a single image (e.g., a single OCT image, an image of a different imaging modality, etc.) to determine or identify one or more tissue types and/or tissue characteristics (or otherwise characterize tissue).
As shown in
Then, by using the column part of each detected area, one or more embodiments of the present disclosure may point, mark, or otherwise indicate in the cross section OCT frames of the pullback where the calcium and/or lipid starts and ends (see e.g., corresponding white lines 42 in the intravascular/OCT images 41, which are also connected to corresponding lines 43 in the top carpet view 40 via the dashed lines in
As aforementioned, once trained, a neural network or networks may use a single image (e.g., a single OCT image, an image of a different imaging modality, etc.) to determine or identify one or more tissue types and/or tissue characteristics (or otherwise characterize tissue).
At least one application embodiment example of using the tissue characterization method of
In one or more embodiments, a model (which, in one or more embodiments, may be software, software/hardware combination, or a procedure that utilizes one or more machine or deep learning algorithms/procedures/processes that has/have been trained on data to make one or more predictions for future, unseen data) has enough resolution to predict and/or evaluate the tissue characterization with sufficient accuracy depending on the application or procedure being performed. The performance of the model may be further improved by subsequently adding more training data and retraining the model to create a new instance of the model with better or optimized performance. For example, additional training data may include data based on user input, where the user may identify or correct the location of a tissue or tissues in an image.
One or more methods, medical imaging devices, Intravascular Ultrasound (IVUS) or Optical Coherence Tomography (OCT) devices, imaging systems, and/or computer-readable storage mediums for evaluating tissue characterization(s) using artificial intelligence may be employed in one or more embodiments of the present disclosure.
In one or more embodiments, an artificial intelligence training apparatus using a neural network or other AI-ready network may include: a memory; one or more processors in communication with the memory, the one or more processors operating to: training a classifier or patch feature extraction and training an AI-classifier (e.g., a ML classifier, a DL classifier, etc.). In one or more embodiments of the present disclosure, an apparatus, a system, or a storage medium may use an AI network, a neural network, or other AI-ready network to perform any of the aforementioned method step(s), including, but not limited to, the steps of
The one or more processors may further operate to use one or more neural networks, convolutional neural networks, and/or recurrent neural networks (or other AI-ready or AI compatible network(s)) to one or more of: load the trained model, select a set of image frames, evaluate the tissue characterization, construct the image (e.g., the carpet view image (CVI)), perform the coregistration, overlay data on the image and/or the intravascular image(s) (e.g., the CVI, the OCT image(s), etc.) and acquire or receive the image data during the pullback operation.
In one or more embodiments, the object, target, or sample may include one or more of the following: a vessel, a target specimen or object, one or more tissues, and a patient (or a target or tissue(s) in the patient).
The one or more processors may further operate to perform the coregistration by co-registering an acquired or received angiography image and an obtained one or more intravascular images, such as, but not limited to. Optical Coherence Tomography (OCT) or Intravascular Ultrasound (IVUS) images or frames, and/or by co-registering the carpet view (CVI) with the one or more intravascular images, such as, but not limited to, Optical Coherence Tomography (OCT) or Intravascular Ultrasound (IVUS) images or frames.
In one or more embodiments, a loaded, trained model may be one or a combination of the following: a random forest(s) model, a Support Vector Machine (SVM) model, a segmentation (classification) model, a segmentation model with pre-processing, a segmentation model with post-processing, an object detection (regression) model, an object detection model with pre-processing, an object detection model with post-processing, a combination of a segmentation (classification) model and an object detection (regression) model, a deep convolutional neural network model, a recurrent neural network model with long short-term memory that can take temporal relationships across images or frames into account, a model using feature pyramid(s) that can take different image resolutions into account, a genetic algorithm that operates to breed multiple models for improved performance, and/or a model using residual learning technique(s).
In one or more embodiments, the one or more processors may further operate to one or more of the following: any of the steps of
One or more embodiments of the present disclosure may use other artificial intelligence technique(s) or method(s) for performing training, for splitting data into different groups (e.g., training group, validation group, test group, etc.), or other artificial intelligence technique(s) or method(s), such as, but not limited to, embodiment(s) as discussed in PCT/US2020/051615, filed on Sep. 18, 2020 and published as WO 2021/055837 A9 on Mar. 25, 2021, and as discussed in U.S. patent application Ser. No. 17/761,561, filed on Mar. 17, 2022, the applications and publications of which are incorporated by reference herein in their entireties. For example, angiography data and/or intravascular data may be used for training, validation, and/or testing as desired. One or more embodiments of a non-transitory computer-readable storage medium storing at least one program for causing a computer to execute a method for training a model using artificial intelligence may be used with any method(s) discussed in the present disclosure, including but not limited to, one or more tissue type and/or characteristic evaluation/determination method(s).
In one or more embodiments of the present disclosure, an OCT image may be formed in a polar coordinate system from A-lines. Each A-line includes much information about the imaged object, such as, but not limited to: clear indications of artifacts from metal objects (e.g., stents, stent struts, guide wires, etc.) like narrow signal width and/or sharp rising and falling edges; significant difference in signal intensity and shape for unobstructed soft tissue compared to the sheath reflection and other artifacts like wide signal width and a gentle falling edge, tissue type and/or tissue characterization, etc. Each A-line represents a cross-sectional 1D sampling of a target, sample, object, etc., such as, but not limited to, a vessel, along a certain view angle. As an imaging probe or device rotates (e.g., rotates about 0 to about 360 degrees, about 180 degrees to about 360 degrees, about 360 degrees, etc.), the corresponding A-lines form the complete two-dimensional (2D) cross-section of the target, sample, object, etc. (e.g., the vessel) in polar coordinates, which is then converted into Cartesian coordinates to form a tomographical-view (tomo-view) image of the cross-section of the target, sample, object, etc. (e.g., the vessel).
In accordance with at least one aspect of the present disclosure and as aforementioned, one or more additional methods for target or object detection of OCT images are provided herein and are discussed in U.S. patent application Ser. No. 16/414,222, filed on May 16, 2019 and published on Dec. 12, 2019 as U.S. Pat. Pub. No. 2019/0374109, the entire disclosure of which is incorporated by reference herein in its entirety. By way of a few examples, pre-processing may include, but is not limited to, one or more of the following steps: (1) smoothening the 2D image in the Polar coordinate (e.g., using a Gaussian filter, as discussed below, etc.), (2) computing vertical and/or horizontal gradients using a Sobel operator (as otherwise discussed below, etc.), and/or (3) computing binary image using an Otsu method. For example, Otsu's method is an automatic image thresholding technique to separate pixels into two classes, foreground and background, and the method minimizes the intraclass variances between two classes and is equivalent to a globally optimal k-means (see e.g., https://en.wikipedia.org/wiki/Otsu %27s_method). One skilled in the art would appreciate that pre-processing methods other than Otsu's method (such as, but not limited to, Jenks optimization method) may be used in addition to or alternatively to Otsu's method in one or more embodiments.
By way of at least one embodiment example of sheath, a Polar coordinate image (e.g., an OCT Polar coordinate image) may include (from the top side to the bottom side, from the top side to the bottom side of an OCT Polar coordinate image, etc.) a sheath area, and a normal field of view (FOV). In one or more embodiments, the lumen area and edge are within the FOV. Because one or more shapes of the sheath may not be a circle (as may be typically assumed) and because the sheath (and, therefore, the sheath shape) may be attached to or overlap with the lumen or guide wire, it may be useful to separate the sheath from the other shapes (e.g., the lumen, the guide wire, tissue(s), etc.) ahead of time.
By way of at least one embodiment example of computing/finding a peak and a major or maximum gradient edge (e.g., for each A-line), soft tissue and other artifacts may be presented on each A-line by one or more peaks with different characteristics, for example, in one or more embodiments of a lumen OCT image(s) (e.g., a normal lumen OCT image). For example, the soft tissue may have a wide bright region beyond the lumen edge, while the artifacts may produce an abrupt dark shadow area beyond the edge. Due to the high-resolution nature of one or more OCT images, transitions between neighbor A-lines may have signals for both peaks. Such signals may allow one or more method embodiments or processes to obtain more accurate locations of the artifact objects and/or the lumen edges. In one or more embodiments, detection of tissue, lumen edges, artifacts, etc. may be performed as discussed in U.S. Pat. Pub. No. 2021/0174125 A1, published on Jun. 10, 2021, the disclosure of which is incorporated by reference herein in its entirety.
Additionally or alternatively, in one or more embodiments, a principal component analysis method and/or a regional covariance descriptor(s) may be used to detect objects, such as tissue(s). Cross-correlation among neighboring images may be used to improve tissue characterization and/or detection result(s). One or more embodiments may employ segmentation based image processing and/or gradient based edge detection to improve detection result(s).
Additionally or alternatively, automatic thresholding (such as the automatic thresholding of step S305 of
where “mean” is the average of the smoothed A-line and “peak” is the maximum value of the smoothed A-line.
As a further example, another approach to find the threshold is to find the average between the max peak and min peak as:
A further alternative approach is to find the threshold based on the max peak as:
Regardless of the approach, the predetermined or determined threshold is used to detect the most significant pulse corresponding to a tissue or tissues in the specific A-line. Any pulse above the threshold is a tissue pulse candidate. The largest pulse among all the candidates in terms of area under the pulse is considered to be the maximum peak (or the “most significant pulse”). The location of the highest peak of the one dimensional gradient signal along the A-line in the vicinity of the maximum peak is used to identify the exact location of the tissue or tissue(s) point in the smoothed A-line.
Placing together all the tissue points thus detected from all the A-lines forms the tissue edge or line as a function of maximum peak locations vs. A-line indices.
One or more methods or algorithms for performing co-registration and/or imaging may be used in one or more embodiments of the present disclosure, including, but not limited to, the methods or algorithms discussed in U.S. Pat. App. No. 62/798,885, filed on Jan. 30, 2019, discussed in U.S. patent application Ser. No. 17/427,052, filed on Jan. 28, 2020, and discussed in U.S. Pat. Pub. No. 2019/0029624, which application(s) and publication(s) are incorporated by reference herein in their entireties.
Such information and other features discussed herein may be applied to other applications, such as, but not limited to, co-registration, other modalities, etc. Indeed, the useful applications of the features of the present disclosure and of the aforementioned applications and patent publications are not limited to the discussed modalities, images, or medical procedures. Additionally, depending on the involved modalities, images, or medical procedures, one or more control bars may be contoured, curved, or have any other configuration desired or set by a user. For example, in an embodiment using a touch screen as discussed herein, a user may define or create the size and shape of a control bar based on a user moving a pointer, a finger, a stylus, another tool, etc. on the touch screen (or alternatively by moving a mouse or other input tool or device regardless of whether a touch screen is used or not).
A computer, such as the console or computer 1200, 1200′, may perform any of the steps (e.g., steps S300-S37 of
In accordance with one or more further aspects of the present disclosure, bench top systems may be utilized for one or more features of the present disclosure, such as, but not limited to, for one or more imaging modalities (such as, but not limited to, angiography, Optical Coherence Tomography (OCT), Multi-modality OCT (MM-OCT), near-infrared auto-fluorescence (NIRAF), near-infrared fluorescence (NIRF), OCT-NIRAF, OCT-NIRF, etc.), and/or for employing one or more additional features discussed herein, including, but not limited to, artificial intelligence processes (e.g., machine or deep learning, residual learning, artificial intelligence (“AI”) co-registration, tissue detection, tissue characterization, etc.) in accordance with one or more aspects of the present disclosure.
The electrical analog signals may be converted to the digital signals to analyze them with a computer, such as, but not limited to, the computer 1200 (see
In one or more embodiments including the deflecting or deflected section 108 (best seen in
In accordance with one or more further aspects of the present disclosure, one or more other systems may be utilized with one or more of the multiple imaging modalities and related method(s) as disclosed herein.
In one or more embodiments, the optical fiber in the catheter 120 operates to rotate inside the catheter 120, and the OCT light and excitation light may be emitted from a side angle of a tip of the catheter 120. After interacting with the object or patient 1o6, the OCT light may be delivered back to an OCT interferometer (e.g., via the circulator 901 of the sample arm 103), which may include the coupler or combiner 903, and combined with the reference beam (e.g., via the coupler or combiner 903) to generate interference patterns. The output of the interferometer is detected with a first detector 107, wherein the first detector 107 may be photodiodes or multi-array cameras, and then may be recorded to a computer (e.g., to the computer 2, the computer 1200 as shown in
Simultaneously or at a different time, the fluorescence intensity may be recorded through a second detector 107 (e.g., a photomultiplier) through a second data-acquisition unit or board (“DAQ2”). The OCT signal and fluorescence signal may be then processed by the computer (e.g., to the computer 2, the computer 1200 as shown in
Detected fluorescence or auto-fluorescence signals may be processed or further processed as discussed in U.S. Pat. App. No. 62/861,888, filed on Jun. 14, 2019, the disclosure of which is incorporated herein by reference in its entirety, as discussed in U.S. patent application Ser. No. 16/898,293, filed Jun. 10, 2020, the disclosure of which is incorporated herein by reference in its entirety, and/or as discussed in U.S. patent application Ser. No. 16/368,510, filed Mar. 28, 2019, the disclosure of which is incorporated herein by reference herein in its entirety.
While not limited to such arrangements, configurations, devices or systems, one or more embodiments of the devices, apparatuses, systems, methods, storage mediums, GUI's, etc. discussed herein may be used with an apparatus or system as aforementioned, such as, but not limited to, for example, the system 20, the system 100, the system 100′, the system 100″, the devices, apparatuses, or systems of
The light source 101 may include a plurality of light sources or may be a single light source. The light source 101 may be a broadband lightsource, and may include one or more of a laser, an organic light emitting diode (OLED), a light emitting diode (LED), a halogen lamp, an incandescent lamp, supercontinuum light source pumped by a laser, and/or a fluorescent lamp. The light source 101 may be any light source that provides light which may then be dispersed to provide light which is then used for imaging, performing control, viewing, changing, emphasizing methods for imaging modalities, constructing or reconstructing image(s) or structure(s), characterizing tissue, and/or any other method discussed herein. The light source 101 may be fiber coupled or may be free space coupled to the other components of the apparatus and/or system 100, 100′, 100″, the devices, apparatuses or systems of
Additionally or alternatively, the one or more detectors 107 may be a linear array, a charge-coupled device (CCD), a plurality of photodiodes or some other method of converting the light into an electrical signal. The detector(s) 107 may include an analog to digital converter (ADC). The one or more detectors may be detectors having structure as shown in one or more of
In accordance with one or more aspects of the present disclosure, one or more methods for performing imaging are provided herein.
The one or more detectors 107 may transmit the digital or analog signals to a processor or a computer such as, but not limited to, an image processor, a processor or computer 1200, 1200′ (see e.g.,
In at least one embodiment, a console or computer 1200, 1200′, a computer 2, any other computer or processor discussed herein, etc. operates to control motions of the RJ via the motion control unit (MCU) 112 or a motor M, acquires intensity data from the detector(s) in the one or more detectors 107, and displays the scanned image (e.g., on a monitor or screen such as a display, screen or monitor 1209 as shown in the console or computer 1200 of any of
The output of the one or more components of any of the systems discussed herein may be acquired with the at least one detector 107, e.g., such as, but not limited to, photodiodes, Photomultiplier tube(s) (PMTs), line scan camera(s), or multi-array camera(s). Electrical analog signals obtained from the output of the system 100, 100′, 100″, and/or the detector(s) 107 thereof, and/or from the devices, apparatuses, or systems of
Unless otherwise discussed herein, like numerals indicate like elements. For example, while variations or differences exist between the systems, such as, but not limited to, the system 20, the system 100, the system 100′, the system 100″, or any other device, apparatus or system discussed herein, one or more features thereof may be the same or similar to each other, such as, but not limited to, the light source 101 or other component(s) thereof (e.g., the console 1200, the console 1200′, etc.). Those skilled in the art will appreciate that the light source 101, the motor or MCU 112, the RJ, the at least one detector 107, and/or one or more other elements of the system 100 may operate in the same or similar fashion to those like-numbered elements of one or more other systems, such as, but not limited to, the devices, apparatuses or systems of
One or more embodiments of the present disclosure may include taking multiple views (e.g., OCT image, ring view, tomo view, anatomical view, etc.), and one or more embodiments may highlight or emphasize NIRF and/or NIRAF. In one or more embodiments, two handles may operate as endpoints that may bound the color extremes of the NIRF and/or NIRAF data in or more embodiments. In addition to the standard tomographic view, the user may select to display multiple longitudinal views. When connected to an angiography system, the Graphical User Interface (GUI) may also display angiography images.
In accordance with one or more aspects of the present disclosure, the aforementioned features are not limited to being displayed or controlled using any particular GUI. In general, the aforementioned imaging modalities may be used in various ways, including with or without one or more features of aforementioned embodiments of a GUI or GUIs. For example, a GUI may show an OCT image with a tool or marker to change the image view as aforementioned even if not presented with a GUI (or with one or more other components of a GUI; in one or more embodiments, the display may be simplified for a user to display set or desired information).
The procedure to select the region of interest and the position of a marker, an angle, a plane, etc., for example, using a touch screen, a GUI (or one or more components of a GUI; in one or more embodiments, the display may be simplified for a user to display the set or desired information), a processor (e.g., processor or computer 2, 1200, 1200′, or any other processor discussed herein) may involve, in one or more embodiments, a single press with a finger and dragging on the area to make the selection or modification. The new orientation and updates to the view may be calculated upon release of a finger, or a pointer. In one or more embodiments, a region of interest and/or a position of the marker may be set or selected automatically using AI features and/or processing features of the present disclosure.
For one or more embodiments using a touch screen, two simultaneous touch points may be used to make a selection or modification, and may update the view based on calculations upon release.
One or more functions may be controlled with one of the imaging modalities, such as the angiography image view or the intravascular image view (e.g., the OCT image view, the IVUS image view, another intravascular imaging modality image view, etc.), to centralize user attention, maintain focus, and allow the user to see all relevant information in a single moment in time.
In one or more embodiments, one imaging modality may be displayed or multiple imaging modalities may be displayed.
One or more procedures may be used in one or more embodiments to select a region of choice or a region of interest for a view. For example, after a single touch is made on a selected area (e.g., by using a touch screen, by using a mouse or other input device to make a selection, etc.), the semi-circle (or other geometric shape used for the designated area) may automatically adjust to the selected region of choice or interest. Two (2) single touch points may operate to connect/draw the region of choice or interest.
There are many ways to compute intensity, viscosity, resolution (including increasing resolution of one or more images), etc., to use one or more imaging modalities, to construct or reconstruct images or structure(s), to detect tissue and/or characterize tissue, and/or related methods for same, discussed herein, digital as well as analog. In at least one embodiment, a computer, such as the console or computer 1200, 1200′, may be dedicated to control and monitor the imaging (e.g., OCT, single mode OCT, multimodal OCT, multiple imaging modalities, IVUS imaging modality, another intravascular imaging modality discussed herein or known to those skilled in the art, etc.) devices, systems, methods and/or storage mediums described herein.
The electric signals used for imaging may be sent to one or more processors, such as, but not limited to, a computer or processor 2 (see e.g.,
Various components of a computer system 1200 are provided in
The I/O or communication interface 1205 provides communication interfaces to input and output devices, which may include a light source, a spectrometer, a microphone, a communication cable and a network (either wired or wireless), a keyboard 1210, a mouse (see e.g., the mouse 1211 as shown in
Any methods and/or data of the present disclosure, such as the methods for performing tissue or object characterization, diagnosis, examination, imaging (including, but not limited to, increasing image resolution, performing imaging using one or more imaging modalities, viewing or changing one or more imaging modalities and related methods (and/or option(s) or feature(s)), etc.), tissue detection, and/or coregistration, for example, as discussed herein, may be stored on a computer-readable storage medium. A computer-readable and/or writable storage medium used commonly, such as, but not limited to, one or more of a hard disk (e.g., the hard disk 1204, a magnetic disk, etc.), a flash memory, a CD, an optical disc (e.g., a compact disc (“CD”) a digital versatile disc (“DVD”), a Blu-ray™ disc, etc.), a magneto-optical disk, a random-access memory (“RAM”) (such as the RAM 1203), a DRAM, a read only memory (“ROM”), a storage of distributed computing systems, a memory card, or the like (e.g., other semiconductor memory, such as, but not limited to, a non-volatile memory card, a solid state drive (SSD) (see SSD 1207 in
In accordance with at least one aspect of the present disclosure, the methods, systems, and computer-readable storage mediums related to the processors, such as, but not limited to, the processor of the aforementioned computer 1200, etc., as described above may be achieved utilizing suitable hardware, such as that illustrated in the figures. Functionality of one or more aspects of the present disclosure may be achieved utilizing suitable hardware, such as that illustrated in
As aforementioned, hardware structure of an alternative embodiment of a computer or console 1200′ is shown in
At least one computer program is stored in the SSD 1207, and the CPU 1201 loads the at least one program onto the RAM 1203, and executes the instructions in the at least one program to perform one or more processes described herein, as well as the basic input, output, calculation, memory writing and memory reading processes.
The computer, such as the computer 2, the computer 1200, 1200′, (or other component(s) such as, but not limited to, the PCU, etc.), etc. may communicate with an MCU, an interferometer, a spectrometer, a detector, etc. to perform imaging, and may reconstruct an image from the acquired intensity data. The monitor or display 1209 displays the reconstructed image, and may display other information about the imaging condition or about an object to be imaged. The monitor 1209 also provides a graphical user interface for a user to operate any system discussed herein. An operation signal is input from the operation unit (e.g., such as, but not limited to, a mouse device 1211, a keyboard 1210, a touch panel device, etc.) into the operation interface 1214 in the computer 1200′, and corresponding to the operation signal the computer 1200′ instructs any system discussed herein to set or change the imaging condition (e.g., improving resolution of an image or images), and to start or end the imaging. A light or laser source and a spectrometer and/or detector may have interfaces to communicate with the computers 1200, 1200′ to send and receive the status information and the control signals.
As shown in
Additionally, unless otherwise specified, the term “subset” of a corresponding set does not necessarily represent a proper subset and may be equal to the corresponding set.
While one or more embodiments of the present disclosure include various details regarding a neural network model architecture and optimization approach, in one or more embodiments, any other model architecture, machine learning algorithm, or optimization approach may be employed. One or more embodiments may utilize hyper-parameter combination(s). One or more embodiments may employ data capture, selection, annotation as well as model evaluation (e.g., computation of loss and validation metrics) since data may be domain and application specific. In one or more embodiments, the model architecture may be modified and optimized to address a variety of computer visions issues (discussed below).
One or more embodiments of the present disclosure may automatically detect (predict a spatial location of) a radiodense OCT marker in a time series of X-ray images to co-register the X-ray images with the corresponding OCT images (at least one example of a reference point of two different coordinate systems). One or more embodiments may use deep (recurrent) convolutional neural network(s), which may improve marker detection, tissue detection, tissue characterization, and image co-registration significantly. One or more embodiments may employ segmentation and/or object/keypoint detection architectures to solve one or more computer vision issues in other domain areas in one or more applications. One or more embodiments employ several novel materials and methods to solve one or more computer vision or other issues (e.g., radiodense OCT marker detection in time series of X-ray images, for instance; tissue detection; tissue characterization; etc.).
One or more embodiments employ data capture and selection. In one or more embodiments, the data is what makes such an application unique and distinguishes this application from other applications. For example, images may include a radiodense marker that is specifically used in one or more procedures (e.g., added to the OCT capsule, used in catheters/probes with a similar marker to that of an OCT marker, used in catheters/probes with a similar or same marker even in a case where the catheters/probes use an imaging modality different from OCT, etc.) to facilitate computational detection of a marker and/or tissue detection, characterization, validation, etc. in one or more images (e.g., X-ray images). One or more embodiments may couple a software device or features (model) to hardware (e.g., an OCT probe, a probe/catheter using an imaging modality different from OCT while using a marker that is the same as or similar to the marker of an OCT probe/catheter, etc.). One or more embodiments may utilize animal data in addition to patient data. Training deep learning may use a large amount of data, which may be difficult to obtain from clinical studies. Inclusion of image data from pre-clinical studies in animals into a training set may improve model performance. Training and evaluation of a model may be highly data dependent (e.g., a way in which frames are selected (e.g., pullback only), split into training/validation/test sets, and grouped into batches as well as the order in which the frames, sets, and/or batches are presented to the model, any other data discussed herein, etc.). In one or more embodiments, such parameters may be more important or significant than some of the model hyper-parameters (e.g., batch size, number of convolution layers, any other hyper-parameter discussed herein, etc.). One or more embodiments may use a collection or collections of user annotations after introduction of a device/apparatus, system, and/or method(s) into a market, and may use post market surveillance, retraining of a model or models with new data collected (e.g., in clinical use), and/or a continuously adaptive algorithm/method(s).
One or more embodiments may employ data annotation. For example, one or more embodiments may label pixel(s) representing a marker or a tissue detection, characterization, and/or validation as well as pixels representing a blood vessel(s) at different phase(s) of a procedure/method (e.g., different levels of contrast due to intravascular contrast agent) of frame(s) acquired during pullback.
One or more embodiments may employ incorporation of prior knowledge. For example, in one or more embodiments, a marker location may be known inside a vessel and/or inside a catheter or probe; a tissue location may be known inside a vessel or other type of target, object, or specimen; etc. As such, simultaneous localization of the vessel and marker may be used to improve marker detection and/or tissue detection, characterization, and/or validation. For example, in a case where it is confirmed that the marker of the probe or catheter, or the catheter or probe, is by or near a target area for tissue detection and characterization, the integrity of the tissue identification/detection and/or characterization for that target area is improved or maximized (as compared to a false positive where a tissue may be detected in an area where the probe or catheter (or marker thereof) is not located). In one or more embodiments, a marker may move during a pullback inside a vessel, and such prior knowledge may be incorporated into the machine learning algorithm or the loss function.
One or more embodiments employ loss (cost) and evaluation function(s)/metric(s). For example, use of temporal information for model training and evaluation may be used in one or more embodiments. One or more embodiments may evaluate a distance between prediction and ground truth per frame as well as consider a trajectory of predictions across multiple frames of a time series.
Application of machine learning may be used in one or more embodiment(s), as discussed in PCT/US2020/051615, filed on Sep. 18, 2020 and published as WO 2021/055837 A9 on Mar. 25, 2021, and as discussed in U.S. patent application Ser. No. 17/761,561, filed on Mar. 17, 2022, the applications and publications of which are incorporated by reference herein in their entireties. For example, at least one embodiment of an overall process of machine learning is shown below:
Based on the testing results, steps i and iii may be revisited in one or more embodiments.
One or more models may be used in one or more embodiment(s) to detect and/or characterize a tissue or tissues, such as, but not limited to, the one or more models as discussed in PCT/US2020/051615, filed on Sep. 18, 2020 and published as WO 2021/055837 A9 on Mar. 25, 2021, and as discussed in U.S. patent application Ser. No. 17/761,561, filed on Mar. 17, 2022, the applications and publications of which are incorporated by reference herein in their entireties. For example, one or more embodiments may use a segmentation model, a regression model, a combination thereof, etc.
For regression model(s), the input may be the entire image frame or frames, and the output may be the centroid coordinates of radiopaque markers (target marker and stationary marker, if necessary/desired) and/or coordinates of a portion of a catheter or probe to be used in determining the tissue detection and/or characterization. Additionally or alternatively, in one or more embodiments, input may comprise or include an entire image frame or frames (e.g., the aforementioned constructed CVI image or frame), and the output may be data regarding high textured areas formed due to the presence of sharp edges in an A-line or A-lines representing calcium in intravascular images as well as dark homogeneous areas representing lipids in the input image frame or frames (e.g., in the CVI image or frame). As shown diagrammatically in
Since the output from a segmentation model, in one or more embodiments, is a “probability” of each pixel that may be categorized as a tissue characterization or tissue identification/determination, post-processing after prediction via the trained segmentation model may be developed to better define, determine, or locate the final coordinate of tissue location (or a marker location where the marker is a part of the catheter) and/or determine the type and/or characteristics of the tissue or tissues. One or more embodiments of a semantic segmentation model may be performed using the One-Hundred Layers Tiramisu method discussed in “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation” to Simon Jégou, et al., Montreal Institute for Learning Algorithms, published Oct. 31, 2017 (https://arxiv.org/pdf/1611.09326.pdf), which is incorporated by reference herein in its entirety. A segmentation model may be used in one or more embodiment, for example, as shown in
In one or more embodiments, hyper-parameters may include, but are not limited to, one or more of the following: Depth (i.e., # of layers), Width (i.e., # of filters), Batch size (i.e., # of training images/step): May be >4 in one or more embodiments, Learning rate (i.e., a hyper-parameter that controls how fast the weights of a neural network (the coefficients of regression model) are adjusted with respect the loss gradient), Dropout (i.e., % of neurons (filters) that are dropped at each layer), and/or Optimizer: for example, Adam optimizer or Stochastic gradient descent (SGD) optimizer. In one or more embodiments, other hyper-parameters may be fixed or constant values, such as, but not limited to, for example, one or more of the following: Input size (e.g., 1024 pixel×1024 pixel, 512 pixel×512 pixel, another preset or predetermined number or value set, etc.), Epochs: 100, 200, 300, 400, 500, another preset or predetermined number, etc. (for additional training, iteration may be set as 3000 or higher), and/or Number of models trained with different hyper-parameter configurations (e.g., 10, 20, another preset or predetermined number, etc.).
One or more features discussed herein may be determined using a convolutional auto-encoder, Gaussian filters, Haralick features, and/or thickness or shape of the sample or object (e.g., the tissue or tissues, a specimen, a patient, a target in the patient, etc.).
One or more embodiments of the present disclosure may use machine learning to determine marker or tissue location; to determine, detect, or evaluate tissue type(s) and/or characteristic(s); to perform coregistration; and/or to perform any other feature discussed herein. Machine learning (ML) is a field of computer science that gives processors the ability to learn, via artificial intelligence. Machine learning may involve one or more algorithms that allow processors or computers to learn from examples and to make predictions for new unseen data points. In one or more embodiments, such one or more algorithms may be stored as software or one or more programs in at least one memory or storage medium, and the software or one or more programs allow a processor or computer to carry out operation(s) of the processes described in the present disclosure.
Similarly, the present disclosure and/or one or more components of devices, systems, and storage mediums, and/or methods, thereof also may be used in conjunction with optical coherence tomography probes. Such probes include, but are not limited to, the OCT imaging systems disclosed in U.S. Pat. Nos. 6,763,261; 7,366,376; 7,843,572; 7,872,759; 8,289,522; 8,676,013; 8,928,889; 9,087,368; 9,557,154; 10,912,462; 9,795,301; and U.S. Pat. No. 9,332,942 to Tearney et al., and U.S. Pat. Pub. Nos. 2014/0276011 and 2017/0135584; and WO 2016/015052 to Tearney et al., and arrangements and methods of facilitating photoluminescence imaging, such as those disclosed in U.S. Pat. No. 7,889,348 to Tearney et al., as well as the disclosures directed to multimodality imaging disclosed in U.S. Pat. No. 9,332,942, and U.S. Patent Publication Nos. 2010/0092389, 2011/0292400, 2012/0101374, 2016/0228097, 2018/0045501 and 2018/0003481, and WO 2016/144878, each of which patents and patent publications are incorporated by reference herein in their entireties. As aforementioned, any feature or aspect of the present disclosure may be used with OCT imaging systems, apparatuses, methods, storage mediums or other aspects or features as discussed in U.S. patent application Ser. No. 16/414,222, filed on May 16, 2019 and published on Dec. 12, 2019 as U.S. Pat. Pub. No. 2019/0374109, the entire disclosure of which is incorporated by reference herein in its entirety.
The present disclosure and/or one or more components of devices, systems, and storage mediums, and/or methods, thereof also may be used in conjunction with OCT imaging systems and/or catheters and catheter systems, such as, but not limited to, those disclosed in U.S. Pat. Nos. 9,869,828; 10,323,926; 10,558,001; 10,601,173; 10,606,064; 10,743,749; 10,884,199; 10,895,692; and 11,175,126 as well as U.S. Patent Publication Nos. 2019/0254506; 2020/0390323; 2021/0121132; 2021/0174125; 2022/0040454; 2022/0044428, and WO2021/055837, each of which patents and patent publications are incorporated by reference herein in their entireties.
Further, the present disclosure and/or one or more components of devices, systems, and storage mediums, and/or methods, thereof also may be used in conjunction with continuum robotic systems and catheters, such as, but not limited to, those described in U.S. Patent Publication Nos. 2019/0105468; 2021/0369085; 2020/0375682; 2021/0121162; 2021/0121051; and 2022-0040450, each of which patents and/or patent publications are incorporated by reference herein in their entireties.
Although the disclosure herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure (and are not limited thereto). It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present disclosure. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application relates, and claims priority, to U.S. Prov. Patent Application Ser. No. 63/594,368, filed Oct. 30, 2023, the disclosure of which is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63594368 | Oct 2023 | US |
