Patent Application
20240144475
The present invention relates to automated techniques for the analysis of capillaries of a subject, particularly though not exclusively a human patient.
Capillaries are the smallest blood vessels in the body measuring less than 20 micrometres in diameter [1]. The exchange of many nutrients with the interstitial fluid occurs here including water, oxygen, and other essential nutrients necessary for the maintenance of cellular metabolism [2], [3]. A network of capillaries constitutes the microcirculation of the body [4]. Studies assessing microcirculation reveal that capillary density and/or flow velocity is altered in several diseases [5]-[17]. Furthermore, some studies claim that changes in microcirculation occur early in disease progression and thus microcirculation monitoring could be used for early detection for various clinical conditions [12, 17].
Determining the density of capillaries allows us to understand the surface area available for the exchange of nutrients [18]. One study concludes that South Asians have a higher mortality rate from cardiovascular diseases compared with Caucasians, and capillary density can be used as an early marker [6]. A second study shows measuring capillary density can be a key component in understanding fibrotic diseases [19]. A study on chronic kidney disease has determined that a reduction in capillary density over time can be used as early detection to prompt timely interventions [20]. Quantifying the velocity of the red blood cells within the capillaries detected can help us understand fluid homeostasis and the transcapillary fluid flux [18]. One study concludes that alterations in the velocity of red blood cells within the capillaries influences the amount of supplies being delivered to the surrounding cells and the time it has to be exchanged across the microvascular wall to the surrounding tissues [21]. Another study shows that an increase in blood flow in the cerebral capillary blood flow can be due to a body response trying to regulate a decrease in oxygen tension in the neurons [22].
A common theme among microvascular analysis articles is the time-consuming, laborious tasks that require clinicians and researchers to manually (or semi-automatically) select capillaries and determine the velocity of the blood flow. Such jobs are strenuous on the eyes and are susceptible to errors and observer variations across different datasets.
On average it takes a trained researcher 20 minutes to analyse a 20-second-long microvascular video [23]. The long analysis time and training requirements are some of the reasons that hinder microvascular microscopy to be integrated into routine clinical practice [24]. Furthermore, manual analysis limits the number of parameters that can be analysed within a microvascular video. Flow velocity and capillary density are assessed, while intra-capillary flow heterogeneity, hematocrit and capillary morphology are not routinely recorded [24].
The Applicant has observed that all the above-mentioned parameters are altered in specific patient groups.
In a series of three meetings organized by international experts in microvascular microscopy it was concluded that an automatic assessment of the microcirculation is required in order to integrate microvascular microscopy into clinical practice [24]. It is an aim of the present invention to provide a method for the automatic assessment of microcirculation in microvascular videos.
Object detection in computer vision is one of the most challenging problems where the aim is to find the area occupied by an object instance in an image [25]. Deep learning is one of the powerful tools which significantly increased the mean average precision in object detection competitions (i.e., VOC 2007-2012, ILSVRC 2013-2017) since its arrival in 2012 [26], [27]. Before deep learning, salient detection (SD) methods were the state of the art in object detection [28]-[30].
The overall aim of the present invention is to lift the burden of the manual job from the professionals and allow them to focus on the method and hypotheses development. The goal is to provide an automated tool for the clinicians to easily quantify the capillaries and classify the erythrocyte velocity in microvascular videos.
When viewed from a first aspect, embodiments of the present invention provide an automated method for analysing capillaries in a plurality of images acquired from a subject, the method comprising the following steps:
This first aspect of the invention extends to a device arranged to carry out automated analysis of capillaries in a plurality of images acquired from a subject, the device comprising:
The first aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out an automated method for analysing capillaries in a plurality of images acquired from a subject, the method comprising the following steps:
The first aspect of the invention further extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out an automated method for analysing capillaries in a plurality of images acquired from a subject, the method comprising the following steps:
Thus it will be appreciated that embodiments of the present invention provide an improved approach for the analysis of capillaries in which capillaries can be detected in images (e.g. microvascular videos) and the velocity of red blood cells can be classified. As outlined in further detail below, embodiments of the present invention may combine deep learning methods with rule-based algorithms and two-frame motion estimation techniques. The approach outlined herein may fully automate the analysis of capillaries, while providing a high degree of accuracy and performance. The method may, in some embodiments, be a method of identifying or monitoring circulatory failure in a subject.
The term ‘subject’ as used herein includes any human or non-human animal subject, including any human or non-human mammal, bird, fish, reptile, amphibian, etc. However, in preferred embodiments the subject is a human mammal, e.g. a human patient.
In some embodiments, the method is a non-invasive method. In other embodiments, the method is an invasive method.
The plurality of images may be acquired as independent images (i.e. ‘stills’), however in some embodiments the plurality of images form a video. In a set of such embodiments, the method comprises acquiring a plurality of videos, wherein the analysis is carried out in respect of said plurality of videos. For example, approximately five videos may be used. The videos may be of any suitable length, however in some particular embodiments, each video may be approximately 20 seconds long.
In a preferred set of embodiments, the images are microscopy images (or videos, as appropriate). In some embodiments in which a microscope probe is used, the microscope may be a computer assisted video microscope (CAVM). The microscopy images may be pre-existing (e.g. generated using a microscope or CAVM), however in some embodiments the step of acquiring the images comprises using a microscope probe to generate said images.
In some such embodiments, the method may comprise positioning the microscope probe on a body surface of the subject. For example, the body surface could be the skin, eye, tongue, nails, or any other internal or external body surface as appropriate. However, in some embodiments, the body surface comprises the subject's skin.
The images acquired may be readily usable for the following capillary analysis steps. However, the Applicant has appreciated that, in some embodiments, it may be advantageous to carry out one or more image enhancement processes on the images, e.g. prior to generating the capillary candidate maps. By performing a dynamic range optimisation of the colours and/or carrying out appropriate image processing techniques, known in the art per se, the image quality may be improved or optimised to enhance performance of the capillary analysis process. In some embodiments, the method further comprises carrying out one or more of:
The above image enhancement steps may be particularly advantageous when the images, e.g. microscopy images, have a large dynamic range of colours and wavelengths. The image enhancement techniques listed above may be applied in an appropriate combination to compensate the data to improve the visualisation for the automated processing algorithm.
The Applicant has appreciated that the images may be subject to some unwanted artefacts from motion that occurred during the capture of those images. For example, where a microscope probe is used (e.g. a handheld microscopy probe), the images captured may be subject to some motion effects due to motion of the probe in use. In some embodiments, the method further comprises carrying out a motion compensation process, e.g. prior to generating the capillary candidate maps. The motion compensation process may comprise removing translation, rotation, and/or depth changes which may, for example, arise due to varying pressure applied to the microscope.
As outlined above, multiple maps are generated according to different processes for each of the images, and these maps are subsequently combined. The step of generating these maps may, in some embodiments, comprise inputting each image to a plurality of pipelines. The step of combining the capillary candidate maps to generate the combined capillary candidate map may be carried out using statistical methods. For example, each of the capillary candidate maps (or the pipelines from which they originate) may have a respective likelihood score, either for the overall map or for each RoI it contains. A maximum likelihood function may be carried out taking each capillary candidate map for a given image as an input where the likelihoods for each potential RoI are combined and compared to some threshold to determine whether, balancing all of the different pipeline's contributions, any given region is an area that might reliably contain a capillary. Different weights may be applied to the different pipelines, where these weights may be fixed or may be determined dynamically.
In some embodiments, a first pipeline is arranged to generate a first capillary candidate map, said first pipeline being arranged to generate an image histogram from each image and determines an optimal pixel value threshold, said first pipeline being further arranged to classify each pixel in said image with a first label if a value of said pixel is less than the determined optimal pixel value threshold, and with a second label if the value of said pixel is equal to or greater than the determined optimal pixel value threshold. The first pipeline may then generate a respective set of RoIs based on the labelled pixels. The determination of the optimal pixel value threshold may be carried out using Otsu's algorithm, known in the art per se. The first and second labels may be pixel values of 0 and 255 respectively, however it will be appreciated that other pixel values could be used. Thus, in some embodiments, the first pipeline carries out a binary threshold that checks if the value of each pixels in a given image are less than the derived number (i.e. the optimal pixel value threshold) and sets it to 0; otherwise, it sets it to 255.
Additionally, or alternatively, in some embodiments, a second pipeline is arranged to generate a second capillary candidate map, said second pipeline being arranged to compare a pixel value of each pixel in each image to a truncation threshold, said second pipeline being further arranged to set the value of each pixel having a pixel value greater than said truncation threshold to said truncation threshold. It will be appreciated that, in accordance with such embodiments, any pixels having values in excess of the maximum allowed value (i.e. the truncation threshold) are ‘cut-off’ or ‘clipped’ at that maximum. The second pipeline may then generate a respective set of RoIs based on the labelled pixels.
Additionally, or alternatively, in some embodiments, a third pipeline is arranged to generate a third capillary candidate map, said third pipeline being arranged to rescale an intensity of the image and to apply a threshold value to said rescaled image according to an adaptive mean. This third pipeline may apply a binary thresholding process based on the threshold value of the third pipeline, such that pixels having a value below the threshold are labelled with a first label (e.g. 0) and pixels having a value equal to or above the threshold are labelled with a second label (e.g. 255). The third pipeline may then generate a respective set of RoIs based on the labelled pixels.
Additionally, or alternatively, in some embodiments, a fourth pipeline is arranged to generate a fourth capillary candidate map, said fourth pipeline being arranged to adjust an image sigmoid using a cut-off and gain and to apply a binary thresholding process. As above, the binary thresholding process may label each pixel according to whether it is below the respective threshold of the fourth pipeline or not. The fourth pipeline may then generate a respective set of RoIs based on the labelled pixels.
Additionally, or alternatively, in some embodiments, a fifth pipeline is arranged to generate a fifth capillary candidate map, said fifth pipeline being arranged to rescale the intensity of the image and to apply a binary threshold. The binary threshold simply checks if the value of the pixel is less than the threshold (i.e. a particular number) and if so, labels that pixel with a first label (e.g. 0); otherwise, it labels it with a second label (e.g. 255). The fifth pipeline may then generate a respective set of RoIs based on the labelled pixels. The binary threshold used for this fifth pipeline may, in some embodiments, may be a predetermined threshold. The appropriate threshold may be determine experimentally.
Additionally, or alternatively, in some embodiments, a sixth pipeline is arranged to generate a sixth capillary candidate map, said sixth pipeline being arranged to detect a movement between subsequent images (i.e. adjacent frames) and to label a region of the image associated with said movement as a region of interest.
It will be appreciated that the terms ‘first’ through ‘sixth’ are used for ease of reference, and embodiments of the present invention are envisaged that utilise any suitable combination and permutation of the above six pipelines, in any suitable subset or superset thereof. However, in a preferred set of embodiments, all six of the above pipelines are provided, and their respective outputs combined.
Each of these pipelines suggests a respective set of RoIs, which are generally different for each pipeline. The RoIs may then be projected back onto the original image. These sets of RoIs may, in some embodiments, be processed using a non-max suppression process to replace overlapping RoIs.
The use of multiple different pipelines is particularly advantageous because the different pipelines generally perform differently with different illumination and different skins types. By combining the outputs of each of the pipelines on each of the images (or videos), the probability of detecting all capillaries can be improved and ideally maximised.
The Applicant has appreciated that one or more of the pipelines may generate one or more false positive RoIs (i.e. areas labelled as being capillaries that do not, in reality, contain a capillary). The first neural network used by embodiments of the present invention acts to combine the RoIs from these pipelines in order to classify whether the RoI contains a capillary.
The first and second neural networks typically comprise different neural networks. Each of the neural networks may be trained using appropriate training data. The first neural network may be trained using validated training data. Thus, in some embodiments, the method comprises generating a validated training data set by manually labelling a plurality of capillaries in a plurality of images and supplying said validated training data set to the first neural network during a training phase. Thus the first neural network can learn from the manually labelled images (which may be labelled by a trained researched) so that it may determine the location of capillaries in a new image the neural network has not seen before.
Thus the first neural network may, in general, be trained using a prior data set comprising pre-labelled capillaries. By training the first neural network using images and/or videos labelled in advance (e.g. by a human being trained to do so), the first neural network can learn patterns so as to better be able to determine from a new data set (i.e. the combined capillary map generated using the newly acquired images taken from the subject), where the capillaries are.
The first neural network may, in some embodiments, comprise a convolutional neural network (CNN). The Applicant has appreciates that CNNs and deep neural networks are particularly well-suited to finding the capillaries features across a set of images and provide significant improvements in the are ability to distinguish capillaries from non-capillaries. Once the RoIs are obtained, CNNs provide high performance both in terms of speed and precision for determining whether or not there is a capillary a given RoI.
The Applicant has appreciated that, if the plurality of images (e.g. video) has been stabilized or if there is no recording motion (e.g. due to hand movement), capillaries can potentially be detected using only a single frame (i.e. a single image). In some particularly advantageous arrangements, all capillaries present may be detected within a single image. In some embodiments, this is achieved by applying a Gaussian model to separate the background of the image from the foreground of the image. The resulting single frame may contain a number of RoIs (e.g. approximately 300 RoI), which are snippets of different parts of the image with different sizes. The first neural network (which, as above, may comprise a CNN) may then classify whether each of these RoIs has a capillary or not. This advantageously eliminates the need to take a mean value across the whole video (as is required by some prior art approaches, known in the art per se) and allows each frame to be handled independent with its own frame-specific values. This also brings us closer to real-time capillary detection since it is not necessary to wait for the whole video to be recorded to get the mean value, but rather we can start detecting capillaries from a single independent frame.
Once the capillaries have been detected in the combined capillary candidate map, the detected capillaries may be passed to a velocity detection stage for the determination of optical flow of the capillaries. The determination of the optical flow may, in some embodiments, comprise applying a Gunnar Farneback algorithm (GFA) to the detected capillaries prior to use of the second neural network. The GFA is a two-frame motion estimation algorithm where approximation is done by calculating the quadratic polynomials and polynomial expansion transform between each frame (i.e. between subsequent ones of the plurality of images). The determined polynomial expansion coefficients are used to derive the displacement fields of the pixels, assuming the pixel intensities are constant between the two frames. The difference in the locations of the pixels becomes the velocity vector value for that capillary.
The velocity vector value of each detected capillary may, in some embodiments, be compared to a velocity vector threshold, wherein a velocity of said detected capillary is classified using the result of said comparison. If the velocity vector value for a given capillary is below the threshold, it may be determined that the capillary has no movement. Conversely, if the velocity vector value for a given capillary is above the threshold, it may be determined that the capillary has movement, which may be constant or intermittent movement.
In some embodiments, only capillaries having a velocity vector value above the velocity vector value threshold are passed to the second neural network. In other words, capillaries classified as having no movement are, at least in some embodiments, not assessed using the second neural network. The Applicant has appreciated that using the second neural network to classify ‘no-movement’ capillaries is superfluous and adds time and computational resource overheads. The GFA algorithm therefore acts as a pre-filter to the second neural network and is able to determine if a capillary is worth passing to the second neural network in near real-time. Conversely, if the velocity vector value is above the threshold, the second neural network may be used to classify the capillary as having either constant or intermittent flow. The direction of flow and intra-capillary flow between frames for that capillary may then be determined.
A stabilizing algorithm may be applied to the frames before passing it to GFA. This stabilizing algorithm may be used to offset any variance in pixel locations arising due to inadvertent movement during the capture of the images, e.g. due to movement of the microscope probe and/or movement of the subject.
The second neural network may, in some embodiments, comprise a deep neural network (DNN). It will be appreciated that a DNN is a neural network having multiple layers between its input layer and output layer, where the successive intermediate layers extract successively higher-level features from the raw input (i.e. the images).
The velocity vector threshold may be set to a suitable value. However, in some embodiments, the velocity vector threshold is between approximately 1.0 and 1.5, optionally between approximately 1.1 and 1.4. In some such embodiments, the velocity vector threshold is approximately equal to 1.2, and may in some such embodiments be equal to 1.2.
Thus, in a particular set of embodiments, if the velocity vector value is under 1.2 the capillary is classified as having no movement; and if the velocity vector is larger than 1.2 it is passed to a deep learning algorithm to classify its velocity as having a constant flow or intermittent flow, and to determine the direction of flow and intra-capillary flow between frames.
Using the GFA together with a DNN to learn the velocity classification from a trained researcher provides significant benefits. In particular, this velocity detection method can be considered as an alternative way to the space-time diagram and manual eye analysis approach, currently used in the art. Advantageously, by using the GFA and DNN, there is no need for a central line to deduce the velocity, thus reducing the steps needed to detect velocity. The GFA may also, in some embodiments, be used to deduce the intra-capillary flow heterogeneity. This makes our method more accurate in tracking red blood cells since we do not singly rely on a central line accurately placed between the frames.
In a set of embodiments, the method comprises performing quality analysis on one or more of the plurality of images to determine whether said images meet a quality threshold. The quality threshold may be determined by analysing the image to determine whether the image quality is sufficient for further processing. This analysis may comprise detecting whether the image is: out of focus (i.e. the quality threshold may comprise an image sharpness threshold); too dark (i.e. the quality threshold may comprise an image brightness threshold); or contains motions too large to be compensated (i.e. the quality threshold may comprise an image motion threshold).
In some embodiments, the area of each detected capillary is calculated based on the area occupied by the pixels associated with said detected capillary. In other words, the total amount of pixels within the capillary's boundary may constitute the area of that particular capillary. We can then proceed to calculate the heterogeneity of the perfusion of the capillary to monitor how much red blood cell flows through the capillary. In this way, we are not only able to detect the capillary and its area, but also proceed to detect how much blood flows through it across time. This can provide the clinician with information of the average capillary hematocrit and its variation over time.
In some embodiments, the parameter comprises one or more of the group comprising:
Such parameters may then be used to assess the subject's microcirculation.
The parameters extracted from the images may be interpreted by a human reader or a machine learning algorithm trained to interpret the results, e.g. based on clinical trials and patient outcomes for a variety of different diseases. In some embodiments, a report comprising the extracted parameters is generated and provided as an output.
The Applicant has appreciated that the arrangement described herein may advantageously detect capillaries using only a single image, using a neural network. Thus, when viewed from a second aspect, embodiments of the present invention provide an automated method for analysing capillaries in an image acquired from a subject, the method comprising the following steps:
This second aspect of the invention extends to a device arranged to carry out automated analysis of capillaries in an image acquired from a subject, the device comprising:
The second aspect of the invention also extends to a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to carry out an automated method for analysing capillaries in an image acquired from a subject, the method comprising the following steps:
The first aspect of the invention further extends to a computer software product comprising instructions that, when executed by a processor, cause the processor to carry out an automated method for analysing capillaries in an image acquired from a subject, the method comprising the following steps:
It will be appreciated that the optional features described hereinabove in respect of embodiments of the first aspect of the invention apply equally to the second aspect of the invention. Unless explicitly stated otherwise, any and all features of any given embodiment of either aspect of the invention described herein may be combined with any other embodiment described herein.
Certain embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which:
An exemplary embodiment of the present invention is described herein and is referred to as ‘CapillaryNet’. For capillary detection, we compare the training time, detection time and accuracy of our method against the state-of-the-art object detection algorithms with Mask R-CNN [31] and capillary annotation as performed by trained researchers. Furthermore, we compare our architecture with other state-of-the-art capillary detection architectures. We show that our approach is more accurate than the state-of-the-art and more efficient than the annotations of the trained researchers. For velocity detection, we benchmark the results of state-of-the-art manual red blood cell flow velocity classification performed by researchers against the velocity classification detected by CapillaryNet.
Mask R-CNN algorithm is one of the top-ranked object detection algorithms [1]. Mask R-CNN with a ResNeXt101-FPN achieved state-of-the-art results on the Coco dataset (80 object categories with 1.5 million object instances) [32] for object segmentation. Mask R-CNN architecture extends Faster R-CNN [33] by adding a branch to predict the segmentation mask of each Region of Interest (RoI) on a pixel-to-pixel basis. The method for object segmentation which is introduced by Mask R-CNN is the ‘RoI align’ which preserves the pixel spatial correspondences. It replaces the quantization from the RoI pooling with bilinear interpolation.
The accuracy of capillary detection between Mask R-CNN and CapillaryNet was calculated based on mean average precision (mAP) and Intersection over Union (IoU) with the labelled data. These are the common evaluation measures used to benchmark object detectors performance in supervised learning algorithms [26], [31].
The input and outputs of CapillaryNet are shown in
The CapillaryNet system 2 carries out processing on these images—as is described in further detail below—and produces several outputs 6. In particular, the CapillaryNet system 2 produces: a bounding box around each capillary 8; an area of the capillary within the bounding box 10; a density of the capillaries in the video 12; a quantification of the capillary hematocrit 14; a measure of the intra-capillary flow between frames 16; the direction of flow in the capillary 18; and a classification of velocity 20. The CapillaryNet system 2 is also able to perform morphology detection 21.
The accuracy of CapillaryNet in capillary detection exceeded that of Mask R-CNN by 25% (92.5% and 67.5% respectively). The total time used to train the CapillaryNet was approximately 1 hour, which is significant less than that needed for the Mask R-CNN approach, which took approximately 120 hours, as shown in Table 1. The algorithms were tested on an Asus GPU 2080Ti 11 GB with 32 GB Ram on an i7-8850H processor.
The reason why Mask R-CNN has a slower training time is because of the sheer number of parameters that needs to be fine-tuned. Mask R-CNN has 63,621,918 trainable parameters while CapillaryNet has 6,786,978 parameters. CapillaryNet has ˜10.67% of the size of Mask R-CNN due to the shallower convolutional neural network (CNN) used. Furthermore, having fewer parameters allows us to detect capillaries much faster and more efficiently allowing us to get closer to near real-time capillary detection with just over 4 frames per second.
The reason why Mask R-CNN was less accurate is due to factors arising from the nature of the skin profile and the properties of the lens. This causes varied illumination on different parts of the skin, blur due to the size of a capillary relative to the image, camera shake detected while recording and occlusion due to hair, stains and other artefacts on the skin, as can be seen in the photograph of
These issues make it challenging for the Mask R-CNN and any other convolutional neural network (CNN) RoI based detector to generalize with accuracy equivalent to the trained researcher manual labelling.
CapillaryNet architecture aims to tackle the challenges posed by the profile of the skin by applying several methods to detect RoIs instead of pure CNNs. These RoI detection methods are computationally less expensive in comparison to the Mask R-CNN RoI detection step, making CapillaryNet faster as shown in Table 1 above. By combining salient detection methods with convolutional neural networks, the approach of the present invention provides the ability to distinguish between capillaries and dirt.
Dobble et. al [34] uses a frame averaging method to remove the plasma and white blood cells gaps within the capillary before using an algorithm to detect capillaries. Using frame averaging can lead to a lower overall density calculation since capillaries with many gaps or insufficient blood flow will be disregarded. Furthermore, Dobble et al [34] removes capillaries that are out of focus, since they consider it to add noise to the frame averaging method. From our experiments with handheld microscopy, the nature of a rounded lens on a typical microscope probe may lead to 40% out of focus images on both edges of the video. It is particularly challenging to have a fully focused video the whole time and some parts can always be out of focus. Therefore, this will further significantly reduce the capillary density values. We compensate for the mentioned drawbacks by making the CNN robust against out-of-focus capillaries by training the CNN on real captured data.
Hilty et. al [23] has a similar flow as Dobble et al [34] with minor tweaks. Hilty et al [23] detect capillaries by first generating a mean image across all frames and then passing the resulting image to two pipelines, firstly classifying vessels of 20-30 μm (diameter) as capillaries, secondly any vessel of up to 400 μm (diameter) as venuels. The capillaries are then passed to a modified curvature-based region detection algorithm [35] to an image that has been stabilized and equalized with an adaptive histogram. The result is a vessel map that contains centrelines across structures that are between 20-30 μm wide. As stated by the authors of the curvature-based region detection algorithm [35], this type of detection is unintelligent and can lead to detecting artefacts such as hair or stains with similar sizes.
Furthermore, due to the profile of the skin challenges stated above, the mean of the images across the whole video is not always the best representation value since different parts of the video might have different lighting or capillaries can be out of the optimal focus. Moreover, videos that have slight motion will have to be completely disregarded since the central line is calculated across all frames instead of per frame.
CapillaryNet takes a different approach to tackle all these challenges. Instead of taking the mean of the image, CapillaryNet applies two independent methods. The first method analyses the frame as an individual and contains 5 steps, the second method analyses the video as a whole and looks at consecutive frames. This means if the video is stabilized or with no recording motion due to hand movement, we can detect all the capillaries present in a single frame only. This is achieved by applying a Gaussian model to separate the background from the foreground. The resulting single frame contains ˜300 RoI which are snippets of different parts of the image with different sizes, and a CNN classifies if this region has a capillary or not. This way we eliminate the need to take a mean value across the whole video and treat each frame uniquely with its own values. This also brings us closer to real-time detection since we do not need to wait for the whole video to be recorded to get the mean value, but rather we can start detecting capillaries from a single independent frame.
Similar to Dobble et al [34], Bezemer et al [38] improves the method by using 2D cross correlation to fill up the blood flow gaps caused by plasma and white blood cells. CapillaryNet tackles this issue by using a well-trained CNN to be able to detect capillaries even if they have plasma and white blood cells gaps.
Tam et al [36] detect capillaries through a semi-automated method which requires the user to select points on the image. The algorithm then decides if there is a capillary present. Conversely, the method of the present invention eliminates the need for user input by automatically detecting if a region has a capillary or not.
Geyman et. al [37] takes more of a manual approach by first using a software to click away the major blood vessels and then applying hardcoded calculations to detect the total number capillaries based on the number of pixels in the region-of-interest. This is quite a manual approach and highly susceptible to observer variations across different datasets. CapillaryNet reduces the observer variations factor by attaching a CNN to classify the RoIs. This CNN is trained on a validated capillary dataset set labelled by trained researchers.
Demir et. al [39] uses CLAHE with a median filter and an adjustable threshold to detect capillaries on the weighted mean of five consecutive frames. However, these methods need to be adjusted accordingly depending on the illumination on the video and thickness of the skin. This introduces a manual job where the user must find the right combination of values for different videos, or the same video with different illumination. CapillaryNet tackles this issue by passing the image into several different independent methods (six, in this case) and a CNN that will be able to determine whether or not the RoI has a capillary based on trained data.
Detecting capillary hematocrit can provide a clinician with information regarding the potential of each capillary to deliver oxygen to the surrounding tissue. To our knowledge, so far, only capillary density and flow velocity was assessed in studies. However, if capillaries have a normal flow and are normally distributed but show a low concentration of red blood cells (i.e. low hematocrit) the oxygen delivery ability of the microcirculation may be compromised. CapillaryNet can detect how much a capillary is filled with red blood cells (hematocrit) over time.
The quantification of red blood cell flow is a more challenging task than capillary detection. Some papers base their red blood cell flow on manual quantification of the red blood cell with different scales [40H43] which is subject to intra-individual variation. In manual quantification each individual vessel receives a score representing the average flow velocity estimated by a researcher. Flow velocity scales vary between publications some researchers classifying flow on a scale from 0 to 2 (absent, intermittent, or continuous flow) [1], others on a scale between 0 to 3 (absent, intermittent, sluggish, or normal flow) [1], while others on a scale from 0 to 5 (no-flow, sluggish, continuous very low, continuous low, continuous high, or brisk flow) [44].
More recent papers use space-time diagrams [23], [38], [45] to quantify red blood cell flow. Space-time diagrams (STD) is a fundamental improvement over the manual eye analysis since it is independent of the individual performing the analysis [34], [42]. However, it comes with its own drawback—it is very sensitive to the slightest movements since it strictly counts on the central line being at the centre of the capillary between all the frames in order to have an accurate space-time diagram. Therefore, if the position or width of the capillary changes between the frames due to camera shake or flow variation, the user must re-calibrate the central line to plot an accurate diagram. This might add an extra task to velocity classification which can be prone to errors and user bias. Furthermore, to construct an accurate central line we are dependent on identifying the exact width and length of the capillaries in the earlier stage. Therefore, capillaries that get out of focus must be disregarded since fitting the central line to plot the STD will not be accurate.
CapillaryNet overcomes the limitation by using the Gunnar Farneback algorithm with deep neural networks to learn the velocity classification from a trained researcher. Thus, our velocity detection method can be considered as an alternative way to the space-time diagram and manual eye analysis which is considered as the gold standard for RBC velocity classification [23]. We do not need the central line to deduce the velocity therefore we reduce the steps needed to detect velocity.
Furthermore, we use the Gunnar Farneback algorithm to deduce the intra-capillary flow heterogeneity. This makes our method more accurate in tracking red blood cells since we do not singly rely on a central line accurately placed between the frames. We benchmarked our velocity detection classification against manually labelled red blood cell flow velocities on capillary videos. Our method had an average accuracy of 88% in classifying the velocity of the capillaries in comparison to the trained researcher, as shown in Table 2.
The average velocity vector calculated by the Gunnar Farneback algorithm for capillaries that had no movement was <˜1.03, and those that had intermittent flow or constant flow was >˜1.21, as can be seen in
By taking these velocity vector values into consideration, we labelled bounding boxes with an average velocity of 1.2 or lower as no movement while the rest was passed on to a deep neural network to classify if it was capillaries with constant or intermittent flow. The accuracy was not closer to 100% since some capillaries flow in 3D (out and into the skin) instead of 2D (across the skin). Capillaries with such flow were misclassified the most.
As can be seen in the graph of
The graph of
Flow velocity within a capillary has been calculated in previous studies as an average value. The Applicant has appreciated that healthy subjects have a relatively homogeneous flow velocity, while in certain patient groups the flow velocity varies significantly throughout the duration of the video (e.g. 20 seconds). Obtaining one average value for each capillary does not allow the clinicians to detect this difference. Therefore, measuring the intra-capillary flow velocity heterogeneity has the potential to be used as an additional marker to detect compromised microcirculation. Clinical studies will need to be conducted in order to elucidate the potential of intra-capillary flow velocity heterogeneity and capillary hematocrit as markers for abnormal microcirculation.
In Table 3 we show how CapillaryNet can calculate and derive the microvascular parameters suggested by Hilty [23] and Ince [24]. In addition, we introduce two new parameters that have not been previously monitored in microvascular videos: capillary hematocrit and intra-capillary flow velocity heterogeneity. CapillaryNet is a unique architecture that combines deep neural networks with salient object detection and two-frame motion estimation techniques to detect capillaries and classify the velocity of red blood cells. Our architecture paves the way to a unified method for near real-time bedside analysis of the microcirculation.
Videos were acquired on human subjects by a hand-held digital microscope (Digital Capillaroscope, Inspectis, Sweden) in the form of videos with a resolution of 1920×1080 at 30 fps for 20 seconds. The videos visualized the nutritive capillaries in the skin papillae in the dorsum region 26 of the hand 28, as shown in
For each subject, a total of four to six videos were collected from neighbouring areas within the region of interest Data was obtained from 25 volunteers. The average age of the subjects was 30 years with a standard deviation of 5 years. A signed consent was obtained from all participants. The study was approved by the Regional Committees for medical and health research ethics of Norway.
To calculate the accuracy of the algorithm for capillary detection, a trained researcher analysed the obtained microvascular videos using in-house software for manual marking of capillaries with rectangular bounding boxes. Capillaries visible in different frames of each video were marked. The labelled bounding boxes were then compared with the algorithm output using the mean average precision (mAP) and Intersection over Union (IoU). The algorithm was trained by extracting the capillaries within the bounding boxes which were labelled by the independent researcher (˜2400 images of nutritive capillaries, ˜2600 images of skin with no capillaries, hair, stains and other artefacts).
The algorithm was trained and validated on ˜70% of the labelled data and then tested on the ˜30% of the labelled data. To calculate the accuracy of the algorithm for velocity detection, we correlated the output of the algorithm with the labelled values of the trained researcher on a scale from 1 to 3 (No or Slow Movement, Intermittent Flow, Constant Flow). The velocity detection deep neural network was trained on 500 videos each with a 30 fps frame rate and validated on 50 videos.
As outlined above, one of the outputs of CapillaryNet—the detected capillaries—is shown in
The first stage, as shown in
The second stage is passing these RoIs to a convolutional neural network part of the CapillaryNet, as shown in the block diagram of
The model architecture consists of three convolutional neural networks separated with a max pooling layer. The output of those neurons is then passed to four neural networks with a dropout rate of 50% to reduce data overfitting. The CNN was optimized using Adam [47] and was trained on ˜50 epochs.
The first pipeline 32 applies an OTSU threshold. The OTSU threshold determines an optimal threshold value from the image histogram. The binary threshold then checks if the value of the pixels in the image are less than the derived number and sets it to zero; otherwise, it sets it to 255. An output image typical of this first pipeline 32 can be seen in
The second pipeline 34 applies a truncated threshold, where if the maximum threshold is higher than the value, then it is truncated with the threshold value. An output image typical of this second pipeline 34 can be seen in
The third pipeline 36 rescales the intensity of the image before and then applies a threshold value according to an adaptive mean. An output image typical of this third pipeline 36 can be seen in
The fourth pipeline 38 adjusts the image sigmoid with a specific cut-off and gain and then applies binary thresholding. An output image typical of this fourth pipeline 38 can be seen in
The fifth pipeline 40 rescales the intensity of the image and then applies a binary threshold like the first pipeline. The binary threshold simply checks if the value of the pixel is less than a certain number and sets it to zero; otherwise, it sets it to 255. An output image typical of this fifth pipeline 40 can be seen in
The sixth pipeline 42 detects movement between adjacent frames and highlights them as RoI. An output image typical of this sixth pipeline 42 can be seen in
Each of these outputs from the pipelines 32, 34, 36, 38, 40, 42 provides a respective capillary candidate map, i.e. it provides a respective image containing RoIs that particular pipeline has indicated may contain a capillary.
The values of all these pipelines have been adjusted and calculated in a trial-and-error manner.
In general, and as outlined above, each of these pipelines 32, 34, 36, 38, 40, 42 will generate a different set of RoIs, which are then projected back onto the original image. These sets of ROIs (i.e. each of the capillary candidate maps) are then passed to a non-max suppression function 44 to replace overlapping RoIs. The overall method (i.e. the combination of these pipelines 32, 34, 36, 38, 40, 42 and the CNN) is illustrated in
Therefore, the outputs of each of the pipelines 32, 34, 36, 38, 40, 42 are combined for each of the videos to maximize the probability of detecting all capillaries, thereby generating a combined capillary candidate map.
The capillary detected 50 is passed to the velocity detection stage of CapillaryNet, where the velocity detection method is illustrated in
A check 56 is made as to whether there is no movement or intermittent flow. If the velocity vector value is under 1.2, the capillary is classified as having no movement, and the direction of flow and the intra-capillary flow between frames is shown 57. If the velocity vector is larger than 1.2, it is passed to a deep learning algorithm 58 to classify its velocity as having a constant flow of intermittent flow. Then the direction of flow and intra-capillary flow between frames is determined using the GFA 60.
Thus the deep neural network 58 is not used to classify no-movement capillaries since doing so may be superfluous, adding time and computational resource overheads. The GFA 54 acts as a pre-filter to the deep neural network 58 and can determine if a capillary is worth passing to the deep neural network 58 in near real-time. The deep neural network where the data was trained on is shown in
The velocity detection algorithm is demonstrated using a publicly available capillary video [49] in
Thus it will be appreciated that embodiments of the present invention provide a fully automated system that can detect capillaries in microvascular videos and classify the velocity of red blood cells by combining deep learning methods with rule-based algorithms and two-frame motion estimation techniques. Thereinafter, the system can quantify the area occupied by a capillary, calculate capillary density, and derive the intra-capillary heterogeneity of flow velocity and quantify capillary hematocrit.
By comparing the training time, detection time and accuracy of the disclosed method against prior art object detection algorithms Mask R-CNN, it can be seen that the method disclosed herein may provide an increase of +25% in accuracy, in half of the detection time.
Furthermore, by comparing the output of embodiments of the present invention to the results of manual capillary detection and velocity quantification performed by trained researchers, it can be seen that CapillaryNet takes 0.2 seconds to detect capillaries in a video whilst it can take up to 2 minutes for a trained researcher to take the capillaries in the video. With an algorithm that can detect capillaries at 4 fps with 92% accuracy, a significant advancement toward real-time capillary detection and area quantification is provided by the approach described herein.
To detect capillaries, CapillaryNet passes the video into a 2-stage process. The first stage aims to detect the regions of interest (RoI) and the second stage passes these RoI into a convolutional neural network to classify if it contains a capillary. If a capillary is detected, the area the capillary occupies is derived by applying a rule-based algorithm. The capillary density is calculated by replicating this step across the whole image. To calculate the content of red blood cells (capillary hematocrit) across time, the area occupied by the capillary in the RoI is calculated across the video. To classify the velocity, we pass the capillary detected to a deep neural network that was trained on researcher classification. Furthermore, we use Farneback algorithm to calculate the intra-capillary heterogeneity across consecutive frames on the video and determine the direction of flow.
Furthermore, the present invention may increase the number of microvascular parameters that can be monitored beyond what is currently possible today with prior art approaches, known in the art per se. Firstly, velocity classification is standardised by using a trained algorithm to detect it. Secondly, the heterogeneity of the flow velocity can be calculated for in a single capillary. Thirdly, the hematocrit of each capillary can be estimated, providing novel information on the potential of each capillary to deliver oxygen to the surrounding tissue.
Thus deep learning techniques are combined with salient object detection and two-frame motion estimation techniques to present a novel method that may automatically:
Those skilled in the art will appreciate that the specific embodiments described herein are merely exemplary and that many variants within the scope of the invention are envisaged.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2103133.1 | Mar 2021 | GB | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/055767 | 3/7/2022 | WO |
