MULTI-ORGAN NUCLEI SEGMENTATION METHOD BASED ON PROMPT LEARNING

Abstract

A multi-organ nuclei segmentation method based on prompt learning is provided. The multi-organ nuclei segmentation method fully mines image information based on text and image information and learns an association between semantic information and a segmentation target, thereby achieving comprehensive learning for target region segmentation. The multi-organ nuclei segmentation method learns a large amount of text and image paired knowledge from six publicly available nucleus datasets based on a clip model to acquire prior knowledge for semantic understanding of nuclei, making a model fully suitable for nuclei segmentation. The constructed model inputs images and text prompts, and utilizes text and image information to achieve nucleus recognition and accurate nuclei segmentation of six different organs, improving computational efficiency. The model can also utilize sufficient text prompts to complete accurate segmentation tasks on some unlabeled datasets, achieving practicality and scalability.

Description

CROSS-REFERENCE TO THE RELATED APPLICATIONS

This application is based upon and claims priority to Chinese Patent Application No. 202311524364.5, filed on Nov. 16, 2023, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the technical field of medical image processing, and in particular to a multi-organ nuclei segmentation method based on prompt learning.

BACKGROUND

At present, most medical image segmentation models focus only on method improvement and exploration for image data. They rarely involve research on multi-modality guided segmentation, and only consider visual factors, lacking comprehensive learning for target region segmentation. In pathological section analysis, multi-organ cell segmentation is recognized as a difficult topic, and the datasets are usually dedicated to specific nuclei segmentation. Therefore, segmentation models are usually constructed based on a partially labeled dataset or a dataset for a specific organ and can only segment the nuclei of a specific organ. More and more segmentation models show excellent performance in single-organ nuclei segmentation, but research on multi-organ nuclei segmentation using a single segmentation model is still limited. To enable a segmentation model to recognize more organs, the model must be retrained to have the segmentation ability for more organs. Therefore, it is a challenge to guide a segmentation model for multi-organ nuclei segmentation based on the multi-modality information including text and image.

SUMMARY

To overcome the above technical deficiencies, the present disclosure provides a method for accurately performing an unsupervised nuclei segmentation task for a specific organ through a sufficient text prompt on an unlabeled dataset.

In order to solve the technical problem, the present disclosure adopts the following technical solution.

A multi-organ nuclei segmentation method based on prompt learning includes the following steps:

• • a) acquiring N brain nuclei images, N kidney nuclei images, N liver nuclei images, N breast nuclei images, N colon nuclei images, and N stomach nuclei images to form a nuclei image-text dataset T,

$\begin{array}{c}T=\{a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{brain}\right]:T_1^{\mathrm{brain}},T_2^{\mathrm{brain}},\dots ,T_i^{\mathrm{brain}},\dots ,T_N^{\mathrm{brain}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{kidney}\right]:T_1^{\mathrm{kidney}},T_2^{\mathrm{kidney}},\dots ,T_i^{\mathrm{kidney}},\dots ,T_N^{\mathrm{kidney}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{liver}\right]:T_1^{\mathrm{liver}},T_2^{\mathrm{liver}},\dots ,T_i^{\mathrm{liver}},\dots ,T_N^{\mathrm{liver}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{breast}\right]:T_1^{\mathrm{breast}},T_2^{\mathrm{breast}},\dots ,T_i^{\mathrm{breast}},\dots ,T_N^{\mathrm{breast}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{colon}\right]:T_1^{\mathrm{colon}},T_2^{\mathrm{colon}},\dots ,T_i^{\mathrm{colon}},\dots ,T_N^{\mathrm{colon}}\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{stomach}\right]:T_1^{\mathrm{stomach}},T_2^{\mathrm{stomach}},\dots ,\\ T_i^{\mathrm{stomach}},\dots ,T_N^{\mathrm{stomach}}\}\end{array},$

where T_i^brain denotes an i-th nuclei image, and a nuclei photo of [brain] is a medical prompt text template for brain; T_i^kidney denotes an i-th kidney nuclei image, and a nuclei photo of [liver] is a medical prompt text template for kidney; T_i^liver denotes an i-th liver nuclei image, and a nuclei photo of [liver] is a medical prompt text template for liver; T_i^breast denotes an i-th breast nuclei image, and a nuclei photo of [breast] is a medical prompt text template for breast; T_i^colon denotes an i-th colon nuclei image, and a nuclei photo of [colon] is a medical prompt text template for colon; and T_i^stomach denotes an i-th stomach nuclei image, and a nuclei photo of [stomach] is a medical prompt text template for stomach;

• • b) inputting the i-th brain nuclei image T_i^brain and the medical prompt text template a nuclei photo of [brain] for brain, the i-th kidney nuclei image T_i^kidney and the medical prompt text template a nuclei photo of [kidney] for kidney, the i-th liver nuclei image T_i^liver and the medical prompt text template a nuclei photo of [liver] for liver, the i-th breast nuclei image T_i^breast and the medical prompt text template a nuclei photo of [breast] for breast, the i-th colon nuclei image T_i^colon and the medical prompt text template a nuclei photo of [colon] for colon, and the i-th stomach nuclei image T_i^stomach and the medical prompt text template a nuclei photo of [stomach] for stomach in the nuclei image-text dataset T into a clip model to acquire an optimized clip model;
  • c) acquiring a brain nuclei images, b kidney nuclei images, c liver nuclei images, d breast nuclei images, e colon nuclei images, and f stomach nuclei images from a nuclei instance segmentation (NuInsSeg) dataset, a+b+c+d+e+f=n; and forming a nuclei image-text dataset Y,

$\begin{array}{c}Y=\{a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{brain}\right]:Y_1^{\mathrm{brain}},Y_2^{\mathrm{brain}},\dots ,Y_i^{\mathrm{brain}},\dots ,Y_a^{\mathrm{brain}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{kidney}\right]:Y_1^{\mathrm{kidney}},Y_2^{\mathrm{kidney}},\dots ,Y_i^{\mathrm{kidney}},\dots ,Y_b^{\mathrm{kidney}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{liver}\right]:Y_1^{\mathrm{liver}},Y_2^{\mathrm{liver}},\dots ,Y_i^{\mathrm{liver}},\dots ,Y_c^{\mathrm{liver}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{breast}\right]:Y_1^{\mathrm{breast}},Y_2^{\mathrm{breast}},\dots ,Y_i^{\mathrm{breast}},\dots ,Y_d^{\mathrm{breast}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{colon}\right]:Y_1^{\mathrm{colon}},Y_2^{\mathrm{colon}},\dots ,Y_i^{\mathrm{colon}},\dots ,Y_e^{\mathrm{colon}}\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{stomach}\right]:Y_1^{\mathrm{stomach}},Y_2^{\mathrm{stomach}},\dots ,\\ Y_i^{\mathrm{stomach}},\dots ,Y_f^{\mathrm{stomach}}\}\end{array},$

where Y_i^brain denotes an i-th brain nuclei image, Y_i^kidney denotes an i-th kidney nuclei image, Y_i^liver denotes an i-th liver nuclei image, Y_i^breast denotes an i-th breast nuclei image, Y_i^colon denotes an i-th colon nuclei image, and Y_i^stomach denotes an i-th stomach nuclei image;

• • d) dividing the nuclei image-text dataset Y into a training set and a test set, and scaling the nuclei image in the training set to 572×572;
  • e) constructing a segmentation network model, including a text module, an image module, and a multilayer perceptron (MILP) module;
  • f) inputting the medical prompt text template in the training set into the text module to acquire a text vector;
  • g) inputting the nuclei image in the training set into the image module of the segmentation network model to acquire a nuclei segmentation result image and a feature vector;
  • h) inputting the feature vector into the MILP module of the segmentation network model to acquire a parameter;
  • i) updating the segmentation network model through the parameter to acquire an updated segmentation network model;
  • j) training the updated segmentation network model to acquire an optimized segmentation network model; and
  • k) inputting the nuclei image in the test set into the optimized segmentation network model to acquire a final segmentation result image.

Preferably, N is 500.

Further, the step a) includes: acquiring the N brain nuclei images from a multi-organ nuclei segmentation (MoNuSeg) dataset and/or a CMP-15 dataset and/or a CMP-17 dataset and/or a nuclei instance segmentation dataset of cryosectioned hematoxylin-eosin-stained histological images (CryonNuSeg) dataset and/or the NuInsSeg dataset; acquiring the N kidney nuclei images from the MoNuSeg dataset and/or a kumar dataset and/or an Irshad dataset and/or the CryonNuSeg dataset and/or a multi-organ nuclei segmentation and classification (MoNuSAC) dataset and/or the NuInsSeg dataset; acquiring the N liver nuclei images from the MoNuSeg dataset and/or the CryonNuSeg dataset and/or a Crowedsourced dataset and/or the kumar dataset and/or the NuInsSeg dataset; acquiring the N breast nuclei images from the MoNuSeg dataset and/or the MoNuSAC dataset and/or a nucleus classification, localization and segmentation (Nucls) dataset and/or a triple negative breast cancer (TNBC) dataset and/or a Janowczyk mCryonNuSeg dataset and/or a Gelasca dataset and/or a Naylor dataset and/or the MoNuSAC dataset and/or a NuInsSeg dataset and/or the kumar dataset; acquiring the N colon nuclei images from a colorectal nuclei segmentation and phenotypes (CoNSeP) dataset and/or a CRCHisto dataset and/or the CryonNuSeg dataset and/or the NuInsSeg dataset and/or the kumar dataset; and acquiring the N stomach nuclei images from a MoNuSeg CryonNuSeg dataset and/or a Wienert dataset and/or the NuInsSeg dataset and/or the kumar dataset.

Preferably, the step d) includes: dividing the nuclei image-text dataset Y into the training set and the test set at a ratio of 7:3.

Further, the step f) includes:

• • f-1) constructing the text module of the segmentation network model, including the optimized clip model;
  • f-2) inputting the medical prompt text template a nuclei photo of [brain] for brain in the training set into the text module to acquire a text vector N_brain, N_brain ∈ R^L×N, where R denotes a real number space, L denotes a length of a text, and N denotes a length of a last word in the text; and expanding, by a torch.unsqueeze function in a PyTorch library of Python, the text vector N_brain by one channel dimension to acquire a text vector N′_brain
  • f-3) inputting the medical prompt text template a nuclei photo of [kidney] for kidney in the training set into the text module to acquire a text vector N_kidney, N_kidney ∈ R^L×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector N_kidney by one channel dimension to acquire a text vector N′_kidney
  • f-4) inputting the medical prompt text template a nuclei photo of [liver] for liver in the training set into the text module to acquire a text vector N_liver N_liver ∈ R^L×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector N_liver, by one channel dimension to acquire a text vector N′_liver,
  • f-5) inputting the medical prompt text template a nuclei photo of [breast] for breast in the training set into the text module to acquire a text vector N_breast N_breast ∈ R^L×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector N_breast by one channel dimension to acquire a text vector N′_breast;
  • f-6) inputting the medical prompt text template a nuclei photo of [colon] for colon in the training set into the text module to acquire a text vector N_colon, N_colon ∈ R^L×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector N_colon on by one channel dimension to acquire a text vector N′_colon; and
  • f-7) inputting the medical prompt text template a nuclei photo of [stomach] for stomach in the training set into the text module to acquire a text vector N_stomach, N_stomach ∈ R L×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector N_stomach by one channel dimension to acquire a text vector N′_stomach.

Further, the step g) includes:

• • g-1) constructing the image module of the segmentation network model, including an image encoder, an image decoder, and a generalizable approximate partitioning (GAP) module;
  • g-2) constructing the image encoder of the image module, including a first CRM, a second CRM, a third CRM, a fourth CRM, and a fifth CRM; constructing the first CRM, the second CRM, the third CRM, and the fourth CRM, each including a first convolutional layer, a first rectified linear unit (ReLU) activation function, a second convolutional layer, a second ReLU activation function, and a max pooling layer in sequence; constructing the fifth CRM, including a first convolutional layer, a first ReLU activation function, a second convolutional layer, and a second ReLU activation function in sequence; inputting the i-th brain nuclei image Y_i^brain in the training set into the first CRM to acquire a feature PE₁^brain, inputting the feature PE₁^brain into the second CRM to acquire a feature PE₂^brain, inputting the feature PE₂^brain into the third CRM to acquire a feature PE₃^brain inputting the feature PE₃^brain into the fourth CRM to acquire a feature PE₄^brain, and inputting the feature PE₄^brain into the fifth CRM to acquire a feature PE₅^brain; inputting the i-th kidney nuclei image Y_i^kidney in the training set into the first CRM to acquire a feature PE₁^kidney, inputting the feature PE₁^kidney into the second CRM to acquire a feature PE₂^kidney, inputting the feature PE₂^kidney into the third CRM to acquire a feature PE₃^kidney, inputting the feature PE₃^kidney into the fourth CRM to acquire a feature PE₄^kidney and inputting the feature PE₄^kidney into the fifth CRM to acquire a feature PE₅^kidney; inputting the i-th liver nuclei image Y_i^liver in the training set into the first CRM to acquire a feature PE₁^liver, inputting the feature PE₁^liver into the second CRM to acquire a feature PE₂^liver, inputting the feature PE₂^liver into the third CRM to acquire a feature PE₃^liver, inputting the feature PE₃^liver into the fourth CRM to acquire a feature PE₄^liver, and inputting the feature PE₄^liver into the fifth CRM to acquire a feature PE₅^liver; inputting the i-th breast nuclei image Y_i^breast in the training set into the first CRM to acquire a feature PE₁^breast, inputting the feature PE₁^breast into the second CRM to acquire a feature PE₂^breast, inputting the feature PE₂^breast into the third CRM to acquire a feature PE₃^breast, inputting the feature PE₃^breast into the fourth CRM to acquire a feature PE₄^breast, and inputting the feature PE₄^breast into the fifth CRM to acquire a feature PE₅^breast; inputting the i-th colon nuclei image Y_i^colon in the training set into the first CRM to acquire a feature PE₁^colon, inputting the feature PE₁^colon into the second CRM to acquire a feature PE₂^colon, inputting the feature PE₂^colon into the third CRM to acquire a feature PE₃^colon inputting the feature PE₃^colon into the fourth CRM to acquire a feature PE₄^colon, and inputting the feature PE₄^colon into the fifth CRM to acquire a feature PE₅^colon; and inputting the i-th stomach nuclei image Y_i^stomach in the training set into the first CRM to acquire a feature PE₁^stomach, inputting the feature PE₁^stomach into the second CRM to acquire a feature PE₂^stomach, inputting the feature PE₂^stomach into the third CRM to acquire a feature PE₃^stomach, inputting the feature PE₃^stomach into the fourth CRM to acquire a feature PE₄^stomach and inputting the feature PE₄^stomach into the fifth CRM to acquire a feature PE₅^stomach.
  • g-3) constructing the image decoder of the image module, including a first GRU module, a second GRU module, a third GRU module, and a fourth GRU module; constructing the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module, each including a first convolutional layer, a first ReLU activation function, a second convolutional layer, a second ReLU activation function, and an upsampling layer in sequence; inputting the feature PE₅^braom into the first CRM to acquire a feature PO₁^brain, inputting the feature PO₁^brain into the second CRM to acquire a feature PO₂^brain,inputting the feature PO₂^brain into the third CRM to acquire a feature PO₃^brain and inputting the feature PO₃^brain into the fourth CRM to acquire a brain nuclei segmentation result image PO₄^brain in inputting the feature PE₅^kidney into the first CRM to acquire a feature PO₁^kidney inputting the feature PO₁^kidney into the second CRM to acquire a feature PO₂^kidney, inputting the feature PO₂^kidney into the third CRM to acquire a feature PO₃^kidney, and inputting the feature PO₃^kidney into the fourth CRM to acquire a kidney nuclei segmentation result image PO₄^kidney; inputting the feature PE₅^liver into the first CRM to acquire a feature PO₁^liver inputting the feature PO₁^liver into the second CRM to acquire a feature PO₂^liver inputting the feature PO₃^liver into the third CRM to acquire a feature PO₃^liver and inputting the feature PO₃^liver into the fourth CRM to acquire a liver nuclei segmentation result image PO₄^liver; inputting the feature PE₅^breast into the first CRM to acquire a feature PO₁^breast, inputting the feature PO₁^breast into the second CRM to acquire a feature PO₂^breast, inputting the feature PO₂^breast into the third CRM to acquire a feature PO₃^breast, and inputting the feature PO₃^breast into the fourth CRM to acquire a breast nuclei segmentation result image PO₄^breast; inputting the feature PE₅^colon into the first CRM to acquire a feature PO₁^colon, inputting the feature PO₁^colon into the second CRM to acquire a feature PO₂^colon inputting the feature PO₁^colon into the third CRM to acquire a feature PO₂^colon, and inputting the feature PO₃^colon into the fourth CRM to acquire a colon nuclei segmentation result image PO₄^colon; and inputting the feature PE₅^stomach into the first CRM to acquire a feature PO₁^stomach, inputting the feature PO₁^stomach into the second CRM to acquire a feature PO₂^stomach inputting the feature PO₂^stomach into the third CRM to acquire a feature PO₃^stomach, and inputting the feature PO₃^stomach into the fourth CRM to acquire a stomach nuclei segmentation result image PO₄^stomach; and
  • g-4) constructing the GAP module of the image module, including a batch normalization (BN) layer, a ReLU activation function, and an adaptive average pooling layer; inputting the feature PE₅^brain into the GAP module to acquire a feature PG^brain inputting the feature PE₅^kidney into the GAP module to acquire a feature PG^kidney inputting the feature PE₅^liver into the GAP module to acquire a feature PG^liver, inputting the feature PE₅^brain into the GAP module to acquire a feature PG^breast, inputting the feature PE₅^colon into the GAP module to acquire a feature PG^colon, and inputting the feature PE₅^stomach into the GAP module to acquire a feature PG^stomach; and concatenating the feature PG^brain and the text vector N′^brain to acquire a feature vector N_mer^brain, concatenating the feature PG^kidney and the text vector N′_kidney to acquire a feature vector N_mer^kidney, concatenating the feature PG^liver and the text vector N′_liver, to acquire a feature vector N_mer^liver concatenating the feature PG^breast and the text vector N′_breast to acquire a feature vector N_mer^breast, concatenating the feature PG^colon and the text vector N′_colon to acquire a feature vector N_mer^colon, and concatenating the feature PG^stomach and the text vector N′_stomach to acquire a feature vector N_mer^stomach.

Further, the step h) includes:

• • h-1) constructing the MLP module of the segmentation network model, including a first convolutional layer, a second convolutional layer, and a third convolutional layer in sequence, where the first convolutional layer, the second convolutional layer, and the third convolutional layer each include a convolutional kernel with a size of 1*1;
  • h-2) inputting the feature vector N_mer^brain into the MLP module to acquire a feature N₁^brain, inputting the feature vector N_mer^kidney into the MLP module to acquire a feature N₁^kidney, inputting the feature vector N_mer^liver into the MLP module to acquire a feature N₁^liver, inputting the feature vector N_mer^breast into the MLP module to acquire a feature N₁^breast, inputting the feature vector N_mer^colon into the MLP module to acquire a feature N₁^colon, and inputting the feature vector N_mer^stomach into the MLP module to acquire a feature N₁^stomach; and
  • h-3) inputting the feature N₁^brain into a Sigmoid function to acquire a parameter θ₁^brain, inputting the feature N₁^kidney into the Sigmoid function to acquire a parameter θ₁^kidney, inputting the feature N₁^liver into the Sigmoid function to acquire a parameter θ₁^liver, inputting the feature N₁^breast into the Sigmoid function to acquire a parameter θ₁^breast, inputting the feature N₁^colon into the Sigmoid function to acquire the parameter θ₁^colon, and inputting the feature N₁^stomach into the Sigmoid function to acquire a parameter θ₁^stomach.

Further, the step i) includes:

• • i-1) performing, by a reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^brain, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by a ReLU activation function in sequence to complete a first update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^brain in in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a first update of the image decoder;
  • i-2) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^kidney in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a second update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^kidney in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a second update of the image decoder;
  • i-3) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^liver, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a third update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^liver, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a third update of the image decoder;
  • i-4) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^breast in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a fourth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^breast, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a fourth update of the image decoder;
  • i-5) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^colon, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a fifth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^colon, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a fifth update of the image decoder; and
  • i-6) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^stomach, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a sixth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ₁^stomach, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a sixth update of the image decoder, thereby acquiring the updated segmentation network model.

Further, the step j) includes: training, by an adaptive moment estimation (Adam) optimizer, the updated segmentation network model through a Dice similarity coefficient (DSC) loss function to acquire the optimized segmentation network model.

Further, the step k) includes:

• • k-1) inputting the i-th brain nuclei image Y_i^brain in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final brain nuclei segmentation result image PO′₄^brain in
  • k-2) inputting the i-th kidney nuclei image Y_i^kidney in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final kidney nuclei segmentation result image PO′₄^kidney;
  • k-3) inputting the i-th liver nuclei image Y_i^liver in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final liver nuclei segmentation result image PO′₄^liver;
  • k-4) inputting the i-th breast nuclei image Y_i^breast in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final breast nuclei segmentation result image PO′₄^breast;
  • k-5) inputting the i-th colon nuclei image Y_i^colon in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final colon nuclei segmentation result image PO′₄^colon; and
  • k-6) inputting the i-th stomach nuclei image Y_i^stomach in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final stomach nuclei segmentation result image PO′₄^brain.

The present disclosure has the following beneficial effects. More and more segmentation models show significant influences in single-organ nuclei segmentation, but research on multi-organ nuclei segmentation using a single segmentation model is still limited and cannot be used in new fields. To solve these problems, the present disclosure proposes a segmentation network model based on text prompt learning. The present disclosure fully mines image information based on text and image information and learns an association between semantic information and a segmentation target, thereby achieving comprehensive learning for target region segmentation. The segmentation network model learns a large amount of text and image paired knowledge from six publicly available nucleus datasets based on the clip model to acquire prior knowledge for semantic understanding of nuclei, making the model fully suitable for nuclei segmentation tasks. The constructed model inputs images and text prompts, and utilizes text and image information to achieve the nucleus recognition and accurate nuclei segmentation of six different organs, improving computational efficiency. The segmentation network model can also utilize sufficient text prompts to complete accurate segmentation tasks on some unlabeled datasets, achieving practicality and scalability.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a structural diagram of a network according to the present disclosure;

FIG. 2 shows sub-images of different organs cropped from MoNuSeg-2018 dataset and segmentation results;

Table 1 shows models comparison results on the MoNuSeg-2018 dataset and the NuInsSeg dataset;

Table 2 shows AJI, F1-score, and Dice on the MoNuSeg-2018 dataset; and

Table 3 shows model comparison results on the MoNuSeg-2018 dataset and state-of-the-art models (SOTAs).

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present disclosure will be described in detail below with reference to FIG. 1.

A multi-organ nuclei segmentation method based on prompt learning includes the following steps.

• • a) N brain nuclei images, N kidney nuclei images, N liver nuclei images, N breast nuclei images, N colon nuclei images, and N stomach nuclei images are acquired to form nuclei image-text dataset T,

$\begin{array}{c}T=\{a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{brain}\right]:T_1^{\mathrm{brain}},T_2^{\mathrm{brain}},\dots ,T_i^{\mathrm{brain}},\dots ,T_N^{\mathrm{brain}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{kidney}\right]:T_1^{\mathrm{kidney}},T_2^{\mathrm{kidney}},\dots ,T_i^{\mathrm{kidney}},\dots ,T_N^{\mathrm{kidney}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{liver}\right]:T_1^{\mathrm{liver}},T_2^{\mathrm{liver}},\dots ,T_i^{\mathrm{liver}},\dots ,T_N^{\mathrm{liver}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{breast}\right]:T_1^{\mathrm{breast}},T_2^{\mathrm{breast}},\dots ,T_i^{\mathrm{breast}},\dots ,T_N^{\mathrm{breast}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{colon}\right]:T_1^{\mathrm{colon}},T_2^{\mathrm{colon}},\dots ,T_i^{\mathrm{colon}},\dots ,T_N^{\mathrm{colon}}\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{stomach}\right]:T_1^{\mathrm{stomach}},T_2^{\mathrm{stomach}},\dots ,\\ T_i^{\mathrm{stomach}},\dots ,T_N^{\mathrm{stomach}}\}\end{array}$

where T_i^brain in denotes an i-th brain nuclei image, and a nuclei photo of [brain] is a medical prompt text template for brain; T_i^kidney denotes an i-th kidney nuclei image, and a nuclei photo of [kidney] is a medical prompt text template for kidney; T_i^liver denotes an i-th liver nuclei image, and a nuclei photo of [liver] is a medical prompt text template for liver; T_i^brain denotes an i-th breast nuclei image, and a nuclei photo of [breast] is a medical prompt text template for breast; T_i^colon denotes an i-th colon nuclei image, and a nuclei photo of [colon] is a medical prompt text template for colon; and T_i^stomach denotes an i-th stomach nuclei image, and a nuclei photo of [stomach] is a medical prompt text template for stomach.

• • b) The i-th brain nuclei image T_i^brain and the medical prompt text template a nuclei photo of [brain] for brain, the i-th kidney nuclei image T_i^kidney and the medical prompt text template a nuclei photo of [kidney] for kidney, the i-th liver nuclei image T_i^liver and the medical prompt text template a nuclei photo of [liver] for liver, the i-th breast nuclei image T_i^breast and the medical prompt text template a nuclei photo of [breast] for breast, the i-th colon nuclei image T_i^colon and the medical prompt text template a nuclei photo of [colon] for colon, and the i-th stomach nuclei image T_i^stomach and the medical prompt text template a nuclei photo of [stomach] for stomach in the nuclei image-text dataset T are input into a clip model to acquire an optimized clip model.
  • c) a brain nuclei images, b kidney nuclei images, c liver nuclei images, d breast nuclei images, e colon nuclei images, and f stomach nuclei images are acquired from a nuclei instance segmentation (NuInsSeg) dataset, a+b+c+d+e+f=n. Nuclei image-text dataset Y is formed,

$\begin{array}{c}Y=\{a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{brain}\right]:Y_1^{\mathrm{brain}},Y_2^{\mathrm{brain}},\dots ,Y_i^{\mathrm{brain}},\dots ,Y_a^{\mathrm{brain}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{kidney}\right]:Y_1^{\mathrm{kidney}},Y_2^{\mathrm{kidney}},\dots ,Y_i^{\mathrm{kidney}},\dots ,Y_b^{\mathrm{kidney}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{liver}\right]:Y_1^{\mathrm{liver}},Y_2^{\mathrm{liver}},\dots ,Y_i^{\mathrm{liver}},\dots ,Y_c^{\mathrm{liver}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{breast}\right]:Y_1^{\mathrm{breast}},Y_2^{\mathrm{breast}},\dots ,Y_i^{\mathrm{breast}},\dots ,Y_d^{\mathrm{breast}};\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{colon}\right]:Y_1^{\mathrm{colon}},Y_2^{\mathrm{colon}},\dots ,Y_i^{\mathrm{colon}},\dots ,Y_e^{\mathrm{colon}}\\ a\mathrm{nuclei}\mathrm{photo}\mathrm{of}\left[\mathrm{stomach}\right]:Y_1^{\mathrm{stomach}},Y_2^{\mathrm{stomach}},\dots ,\\ Y_i^{\mathrm{stomach}},\dots ,Y_f^{\mathrm{stomach}}\}\end{array},$

where Y_i^brain denotes an i-th brain nuclei image, Y_i^kidney denotes an i-th kidney nuclei image, Y_i^liver denotes an i-th liver nuclei image, Y_i^breast denotes an i-th breast nuclei image, Y_i^colon denotes an i-th colon nuclei image, and Y_i^stomach denotes an i-th stomach nuclei image.

• • d) The nuclei image-text dataset Y is divided into a training set and a test set, and scaling the nuclei image in the training set to 572×572.
  • e) A segmentation network model is constructed, including a text module, an image module, and a multilayer perceptron (MLLP) module.
  • f) The medical prompt text template in the training set is input into the text module to acquire a text vector.
  • g) The nuclei image in the training set is input into the image module of the segmentation network model to acquire a nuclei segmentation result image and a feature vector.
  • h) The feature vector is input into the MLP module of the segmentation network model to acquire a parameter.
  • i) The segmentation network model is updated through the parameter to acquire an updated segmentation network model.
  • j) The updated segmentation network model is trained to acquire an optimized segmentation network model.
  • k) The nuclei image in the test set is input into the optimized segmentation network model to acquire a final segmentation result image.

The present disclosure provides a multi-organ nuclei segmentation method based on prompt learning, which can segment multiple different types of organ nuclei through text and image information. In addition, the present disclosure can also accurately complete the unsupervised segmentation task of specific organ nuclei on an unlabeled dataset through sufficient text prompts.

In an embodiment of the present disclosure, N is 500. That is, there are 500 nuclei images for each organ.

In an embodiment of the present disclosure, in the step a), the N brain nuclei images are acquired from a multi-organ nuclei segmentation (MoNuSeg) dataset and/or a CMP-15 dataset and/or a CMP-17 dataset and/or a nuclei instance segmentation dataset of cryosectioned hematoxylin-eosin-stained histological images (CryonNuSeg) dataset and/or the NuInsSeg dataset. The N kidney nuclei images are acquired from the MoNuSeg dataset and/or a kumar dataset and/or an Irshad dataset and/or the CryonNuSeg dataset and/or a multi-organ nuclei segmentation and classification (MoNuSAC) dataset and/or the NuInsSeg dataset. The N liver nuclei images are acquired from the MoNuSeg dataset and/or the CryonNuSeg dataset and/or a Crowedsourced dataset and/or the kumar dataset and/or the NuInsSeg dataset. The N breast nuclei images are acquired from the MoNuSeg dataset and/or the MoNuSAC dataset and/or a nucleus classification, localization and segmentation (Nucls) dataset and/or a triple negative breast cancer (TNBC) dataset and/or a Janowczyk mCryonNuSeg dataset and/or a Gelasca dataset and/or a Naylor dataset and/or the MoNuSAC dataset and/or a NuInsSeg dataset and/or the kumar dataset. The N colon nuclei images are acquired from a colorectal nuclei segmentation and phenotypes (CoNSeP) dataset and/or a CRCHisto dataset and/or the CryonNuSeg dataset and/or the NuInsSeg dataset and/or the kumar dataset. The N stomach nuclei images are acquired from a MoNuSeg CryonNuSeg dataset and/or a Wienert dataset and/or the NuInsSeg dataset and/or the kumar dataset.

In an embodiment of the present disclosure, in the step d), the nuclei image-text dataset Y is divided into the training set and the test set at a ratio of 7:3.

In an embodiment of the present disclosure, the step f) is as follows.

• • f-1) The text module of the segmentation network model is constructed, including the optimized clip model.
  • f-2) The medical prompt text template a nuclei photo of [brain] for brain in the training set is input into the text module to acquire text vector N_brain, N_brain ∈ R^L×N, where R denotes a real number space, L denotes a length of a text, and N denotes a length of a last word in the text. The text vector N_brain is expanded by one channel dimension by a torch.unsqueeze function in a PyTorch library of Python to acquire text vector N′_brain.
  • f-3) The medical prompt text template a nuclei photo of [kidney] for kidney in the training set is input into the text module to acquire text vector N_kidney, N_kidney ∈ R^L×N The text vector N_kidney is expanded by one channel dimension by the torch.unsqueeze function in the PyTorch library of Python to acquire text vector N′_kidney
  • f-4) The medical prompt text template a nuclei photo of [liver] for liver in the training set is input into the text module to acquire text vector N_liver, N_liver ∈R^L×N. The text vector N_liver is expanded by one channel dimension by the torch.unsqueeze function in the PyTorch library of Python to acquire text vector N′_liver.
  • f-5) The medical prompt text template a nuclei photo of [breast] for breast in the training set is input into the text module to acquire text vector N_breast, N_breast ∈R^L×N. The text vector N_breast is expanded by one channel dimension by the torch.unsqueeze function in the PyTorch library of Python to acquire text vector N′_breast.
  • f-6) The medical prompt text template a nuclei photo of [colon] for colon in the training set is input into the text module to acquire text vector N_colon, N_colon ∈R^L×N. The text vector N_colon is expanded by one channel dimension by the torch.unsqueeze function in the PyTorch library of Python to acquire text vector N′_colon.
  • f-7) The medical prompt text template a nuclei photo of [stomach] for stomach in the training set is input into the text module to acquire text vector N_stomach, N_stomach ∈ R^L×N. The text vector N_stomach is expanded by one channel dimension by the torch.unsqueeze function in the PyTorch library of Python to acquire text vector N′_stomach.

In an embodiment of the present disclosure, the step g) is as follows.

• • g-1) The image module of the segmentation network model is constructed, including an image encoder, an image decoder, and a generalizable approximate partitioning (GAP) module.
  • g-2) The image encoder of the image module is constructed, including a first CRM, a second CRM, a third CRM, a fourth CRM, and a fifth CRM. The first CRM, the second CRM, the third CRM, and the fourth CRM are constructed, each including a first convolutional layer, a first rectified linear unit (ReLU) activation function, a second convolutional layer, a second ReLU activation function, and a max pooling layer in sequence. The fifth CRM is constructed, including a first convolutional layer, a first ReLU activation function, a second convolutional layer, and a second ReLU activation function in sequence. The i-th brain nuclei image y_i^brain in the training set is input into the first CRM to acquire feature PE₁^brain, Y_i^brain ∈ R^C×H×W, where C, H, and W denote a channel number, a height, and a width of the image, respectively. The feature PE₁^brain in is input into the second CRM to acquire feature PE₂^brain in the feature PE₂^brain is input into the third CRM to acquire feature PE₃^brain, the feature PE₃^brain is input into the fourth CRM to acquire feature PE₄^brain, and the feature PE₄^brain is input into the fifth CRM to acquire feature PE₅^brain. The i-th kidney nuclei image Y_i^kidney in the training set is input into the first CRM to acquire feature PE₁^kidney, Y_i^kidney ∈ R^C×H×W, the feature PE₁^kidney is input into the second CRM to acquire feature PE₂^kidney, the feature PE₂^kidney is input into the third CRM to acquire feature PE₃^kidney the feature PE₃^kidney is input into the fourth CRM to acquire feature PE₄^kidney, and the feature PE₄^kidney is input into the fifth CRM to acquire feature PE₅^kidney. The i-th liver nuclei image Y_i^liver in the training set is input into the first CRM to acquire feature PE₁^liver, Y_i^liver ∈ R^C×H×W inputting the feature PE₁^liver is input into the second CRM to acquire feature PE₂^liver, the feature PE₂^liver is input into the third CRM to acquire feature PE₃^liver, the feature PE₃^liver is input into the fourth CRM to acquire feature PE₄^liver, and the feature PE₄^liver is input into the fifth CRM to acquire feature PE₅^liver. The i-th breast nuclei image Y_i^breast in the training set is input into the first CRM to acquire feature PE₁^breast, Y_i^breast ∈ R^C×H×W, the feature PE₁^breast is input into the second CRM to acquire feature PE₂^breast, the feature PE₂^breast is input into the third CRM to acquire feature PE₁^breast, the feature PE₃^breast is input into the fourth CRM to acquire feature PE₄^breast, and the feature PE₄^breast is input into the fifth CRM to acquire feature PE₅^breast. The i-th colon nuclei image Y_i^colon in the training set is input into the first CRM to acquire feature PE₁^colon, Y_i^colon ∈ R^C×H×W, the feature PE₁^colon is input into the second CRM to acquire feature PE₂^colon the feature PE₂^colon is input into the third CRM to acquire feature PE₃^colon, the feature PE₃^colon is input into the fourth CRM to acquire feature PE₄^colon, and the feature PE₄^colon is input into the fifth CRM to acquire feature PE₅^colon. The i-th stomach nuclei image Y_i^stomach in the training set is input into the first CRM to acquire feature PE₁^stomach, Y_i^stomach ∈ R^C×H×W, the feature PE₁^stomach is input into the second CRM to acquire feature PE₂^stomach,the feature PE₂^stomach is input into the third CRM to acquire feature PE₃^stomach, the feature PE₃^stomach is input into the fourth CRM to acquire feature PE₄^stomach, and the feature PE₄^stomach is input into the fifth CRM to acquire feature PE₅^stomach.
  • g-3) The image decoder of the image module is constructed, including a first GRU module, a second GRU module, a third GRU module, and a fourth GRU module. The first GRU module, the second GRU module, the third GRU module, and the fourth GRU module are constructed, each including a first convolutional layer, a first ReLU activation function, a second convolutional layer, a second ReLU activation function, and an upsampling layer in sequence. The feature PE₅^brain is input into the first CRM to acquire feature PO₁^brain, the feature PO₁^brain is input into the second CRM to acquire feature PO₂^brain, the feature PO₂^brain is input into the third CRM to acquire feature PO₃^brain, and the feature PO₃^brainis input into the fourth CRM to acquire brain nuclei segmentation result image PO₄^brain. The feature PE₅^kidney is input into the first CRM to acquire feature PO₁^kidney, the feature PO₁^kidney is input into the second CRM to acquire feature PO₂^kidney, the feature PO₂^kidney is input into the third CRM to acquire feature PO₃^kidney, and the feature PO₃^kidney is input into the fourth CRM to acquire kidney nuclei segmentation result image PO₄^kidney. The feature PE₅^liver is input into the first CRM to acquire feature PO₁^liver the feature PO₁^liver is input into the second CRM to acquire feature PO₂^liver, the feature PO₂^liver is input into the third CRM to acquire feature PO₃^liver, and the feature PO₃^liver is input into the fourth CRM to acquire liver nuclei segmentation result image PO₄^liver. The feature PE₅^breast is input into the first CRM to acquire feature PO₁^breast, the feature PO₁^breast is input into the second CRM to acquire feature PO₂^breast, the feature PO₂^breast is input into the third CRM to acquire feature PO₃^breast, and the feature PO₃^breast is input into the fourth CRM to acquire breast nuclei segmentation result image PO₄^breast. The feature PE₅^colon is input into the first CRM to acquire feature PO₁^colon, the feature PO₁^colon is input into the second CRM to acquire feature PO₂^colon, the feature PO₂^colon is input into the third CRM to acquire feature PO₃^colon, and the feature PO₃^colon is input into the fourth CRM to acquire colon nuclei segmentation result image PO₄^colon. The feature PE₅^stomach is input into the first CRM to acquire feature PO₁^stomach, the feature PO₁^stomach is input into the second CRM to acquire feature PO₂^stomach, the feature PO₂^stomach is input into the third CRM to acquire feature PO₃^stomach,and the feature PO₃^stomach is input into the fourth CRM to acquire stomach nuclei segmentation result image PO₄^stomach.
  • g-4) The GAP module of the image module is constructed, including a batch normalization (BN) layer, a ReLU activation function, and an adaptive average pooling layer. The feature PE₅^brain is input into the GAP module to acquire feature PG^brain, the feature PE₅^kidney is input into the GAP module to acquire feature PG^kidney, the feature PE₅^liver is input into the GAP module to acquire feature PG^liver, the feature PE₅^breast is input into the GAP module to acquire feature PG^breast, the feature PE₅^colon is input into the GAP module to acquire feature PG^colon, and the feature PE₅^stomach is input into the GAP module to acquire feature PG^stomach. The feature PG^brain and the text vector N′_brain are concatenated to acquire feature vector N_mer^brain. The feature PG^kidney and the text vector N′_kidney are concatenated to acquire feature vector N_mer^kidney. The feature PG^liver and the text vector N′_liver are concatenated to acquire feature vector N_mer^liver. The feature PG^breast and the text vector N′_breast, are concatenated to acquire feature vector N_mer^breast. The feature PG^colon and the text vector N′_colon are concatenated to acquire feature vector N_mer^colon. The feature PG^stomach and the text vector N′_stomach are concatenated to acquire feature vector N_mer^stomach.

In an embodiment of the present disclosure, the step h) is as follows.

• • h-1) The MLP module of the segmentation network model is constructed, including a first convolutional layer, a second convolutional layer, and a third convolutional layer in sequence, where the first convolutional layer, the second convolutional layer, and the third convolutional layer each include a convolutional kernel with a size of 1*1.
  • h-2) The feature vector N_mer^brain in is input into the MLP module to acquire feature N₁^brain, the feature vector N_mer^kidney is input into the MLP module to acquire feature N₁^kidney, the feature vector N_mer^liver is input into the MLP module to acquire feature N₁^liver, the feature vector N_mer^breast is input into the MLP module to acquire feature N₁^breast, the feature vector N_mer^colon is input into the MLP module to acquire feature N₁^colon, and the feature vector N_mer^stomach is input into the MLP module to acquire feature N₁^stomach.
  • h-3) The feature N₁^brain is input into a Sigmoid function to acquire parameter θ₁^brain, the feature N₁^kidney is input is input into the Sigmoid function to acquire parameter θ₁^kidney, the feature N₁^liver is input is input into the Sigmoid function to acquire parameter θ₁^liver, the feature N₁^breast is input is input into the Sigmoid function to acquire parameter θ₁^breast, the feature N₁^colon is input into the Sigmoid function to acquire the parameter θ₁^colon, and the feature N₁^stomach is input is input into the Sigmoid function to acquire parameter θ₁^stomach.

In an embodiment of the present disclosure, the step i) is as follows.

• • i-1) A reshape operation is performed by a reshape function in the PyTorch library of Python, through the parameter θ₁^brain, in in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by a ReLU activation function are performed in sequence to complete a first update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^brain, brain in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a first update of the image decoder.
  • i-2) A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^kidney, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a second update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^kidney, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a second update of the image decoder.
  • i-3) A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^liver, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a third update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^liver, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a third update of the image decoder.
  • i-4) A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^breast, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a fourth update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^breast, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a fourth update of the image decoder.
  • i-5) A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^colon, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a fifth update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^colon, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a fifth update of the image decoder.
  • i-6) A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^stomach, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a sixth update of the image encoder. A reshape operation is performed by the reshape function in the PyTorch library of Python, through the parameter θ₁^stomach, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder. A convolution, and an operation by the ReLU activation function are performed in sequence to complete a sixth update of the image decoder, thereby acquiring the updated segmentation network model.

In an embodiment of the present disclosure, in the step j), the updated segmentation network model is trained by an adaptive moment estimation (Adam) optimizer through a Dice similarity coefficient (DSC) loss function to acquire the optimized segmentation network model.

In an embodiment of the present disclosure, the step k) is as follows.

• • k-1) The i-th brain nuclei image Y_i^brain in in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final brain nuclei segmentation result image PO′₄^brain.
  • k-2) The i-th kidney nuclei image Y_i^kidney in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final kidney nuclei segmentation result image PO′₄^kidney.
  • k-3) The i-th liver nuclei image Y_i^liver in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final liver nuclei segmentation result image PO′₄^liver.
  • k-4) The i-th breast nuclei image Y_i^breast in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final breast nuclei segmentation result image PO′₄^breast.
  • k-5) The i-th colon nuclei image Y_i^colon in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final colon nuclei segmentation result image PO′₄^colon.
  • k-6) The i-th stomach nuclei image Y_i^stomach in the training set is input into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire final stomach nuclei segmentation result image PO′₄^stomach.

Finally, it should be noted that the above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art may still modify the technical solutions described in the foregoing embodiments, or equivalently substitute some technical features thereof. Any modification, equivalent substitution, improvement, etc. within the spirit and principles of the present disclosure shall fall within the scope of protection of the present disclosure.

The proposed model is qualitatively analyzed. As shown in FIG. 2, the seven sub-images were cropped from seven different MoNuSeg-2018 test images of seven different organs, where three organs (bladder, lung, prostate) not included in the training test and a validation set. Original images and nuclei segmentation images are listed in rows.

FIG. 2 shows that the proposed model can successfully segment individual nuclei, indicating that the proposed model has strong generalization ability. The overlap of certain nuclei and the diversity of tissue types, nucleus appearance, and H & E staining are not considered, and there are some false positive cases (over-segmentation) and false negative cases (under-segmentation), but their number is not significant relative to the total number of true positive cases.

For quantitative analysis, Table 1 lists the average values of aggregated Jaccard index (AJI), F1-score, and Dice for two test datasets. Some images in the NuInsSeg dataset show the nuclei of organs that are not present in the training set and the validation set. The test results are similar to those on the MoNuSeg-2018 dataset, indicating that the proposed model has strong generalization ability.

Table 2 shows the AJI, F1-score, and Dice of the proposed model in segmenting six types of organs in the MoNuSeg-2018 dataset. These values are relatively stable across different organ segmentation tasks. The stability of these scores is positive for the performance of the model, indicating that the model can maintain high consistency and stability in different organ segmentation tasks, which suggests that the model has strong applicability.

Table 3 shows that the proposed model achieves good performance in the nuclei segmentation task. Specifically, the model achieves a Dice score of 0.8311 and an AJI score of 0.6413 on the MoNuSeg dataset, showing an obvious improvement in the performance of the model compared to that on Hover-Net and CIA-Net. It should be noted that the proposed model is designed for multi-organ segmentation and has strong generalization ability. However, if it encounters an unfamiliar organ, it needs to be retrained.

Claims

1. A multi-organ nuclei segmentation method based on a prompt learning, comprising the following steps: a) acquiring N brain nuclei images, N kidney nuclei images, N liver nuclei images, N breast nuclei images, N colon nuclei images, and N stomach nuclei images to form a nuclei image-text dataset T,

2. The multi-organ nuclei segmentation method based on the prompt learning according to claim 1, wherein N is 500.

3. The multi-organ nuclei segmentation method based on the prompt learning according to claim 1, wherein the step a) comprises: acquiring the N brain nuclei images from a multi-organ nuclei segmentation (MoNuSeg) dataset and/or a CMP-15 dataset and/or a CMP-17 dataset and/or a nuclei instance segmentation dataset of cryosectioned hematoxylin-eosin-stained histological images (CryonNuSeg) dataset and/or the NuInsSeg dataset;acquiring the N kidney nuclei images from the MoNuSeg dataset and/or a kumar dataset and/or an Irshad dataset and/or the CryonNuSeg dataset and/or a multi-organ nuclei segmentation and classification (MoNuSAC) dataset and/or the NuInsSeg dataset;acquiring the N liver nuclei images from the MoNuSeg dataset and/or the CryonNuSeg dataset and/or a Crowedsourced dataset and/or the kumar dataset and/or the NuInsSeg dataset;acquiring the N breast nuclei images from the MoNuSeg dataset and/or the MoNuSAC dataset and/or a nucleus classification, localization and segmentation (Nucls) dataset and/or a triple negative breast cancer (TNBC) dataset and/or a Janowczyk mCryonNuSeg dataset and/or a Gelasca dataset and/or a Naylor dataset and/or the MoNuSAC dataset and/or a NuInsSeg dataset and/or the kumar dataset;acquiring the N colon nuclei images from a colorectal nuclei segmentation and phenotypes (CoNSeP) dataset and/or a CRCHisto dataset and/or the CryonNuSeg dataset and/or the NuInsSeg dataset and/or the kumar dataset; andacquiring the N stomach nuclei images from a MoNuSeg CryonNuSeg dataset and/or a Wienert dataset and/or the NulnsSeg dataset and/or the kumar dataset.

4. The multi-organ nuclei segmentation method based on the prompt learning according to claim 1, wherein the step d) comprises: dividing the nuclei image-text dataset Y into the training set and the test set at a ratio of 7:3.

5. The multi-organ nuclei segmentation method based on the prompt learning according to claim 1, wherein the step f) comprises: f-1) constructing the text module of the segmentation network model, comprising the optimized clip model;f-2) inputting the medical prompt text template a nuclei photo of [brain] for brain in the training set into the text module to acquire a text vector Nbrain, Nbrain ∈ RL×N, wherein R denotes a real number space, L denotes a length of a text, and N denotes a length of a last word in the text; and expanding, by a torch.unsqueeze function in a PyTorch library of Python, the text vector Nbrain by one channel dimension to acquire a text vector N′brain;f-3) inputting the medical prompt text template a nuclei photo of [kidney] for kidney in the training set into the text module to acquire a text vector Nkidney, Nkidney ∈ RL×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector Nkidney by one channel dimension to acquire a text vector N′kidney;f-4) inputting the medical prompt text template a nuclei photo of [liver] for liver in the training set into the text module to acquire a text vector NLiver, Nliver ∈RL×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector Nliver by one channel dimension to acquire a text vector N′liver;f-5) inputting the medical prompt text template a nuclei photo of [breast] for breast in the training set into the text module to acquire a text vector Nbreast, Nbreast ∈ RL×NN; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector Nbreast by one channel dimension to acquire a text vector N′breast;f-6) inputting the medical prompt text template a nuclei photo of [colon] for colon in the training set into the text module to acquire a text vector Ncolon, Ncolon ∈ RL×N, and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector Ncolon by one channel dimension to acquire a text vector N′colon; andf-7) inputting the medical prompt text template a nuclei photo of [stomach] for stomach in the training set into the text module to acquire a text vector Nstomach; Nstomach ∈ RL×N; and expanding, by the torch.unsqueeze function in the PyTorch library of Python, the text vector Nstomach by one channel dimension to acquire a text vector Nstomach.

6. The multi-organ nuclei segmentation method based on the prompt learning according to claim 5, wherein the step g) comprises: g-1) constructing the image module of the segmentation network model, comprising an image encoder, an image decoder, and a generalizable approximate partitioning (GAP) module;g-2) constructing the image encoder of the image module, comprising a first CRM, a second CRM, a third CRM, a fourth CRM, and a fifth CRM;constructing the first CRM, the second CRM, the third CRM, and the fourth CRM, each comprising a first convolutional layer, a first rectified linear unit (ReLU) activation function, a second convolutional layer, a second ReLU activation function, and a max pooling layer in sequence;constructing the fifth CRM, comprising a first convolutional layer, a first ReLU activation function, a second convolutional layer, and a second ReLU activation function in sequence;inputting the i-th brain nuclei image Yibrain in the training set into the first CRM to acquire a feature PE1brain, inputting the feature PE1brain into the second CRM to acquire a feature PE2brain, inputting the feature PE2brain into the third CRM to acquire a feature PE3brain, inputting the feature PE3brain into the fourth CRM to acquire a feature PE4brain, and inputting the feature PE4brain in into the fifth CRM to acquire a feature PE5brain;inputting the i-th kidney nuclei image Yikidney in the training set into the first CRM to acquire a feature PE1kidney, inputting the feature PE1kidney into the second CRM to acquire a feature PE2kidney, inputting the feature PE2kidney into the third CRM to acquire a feature PE3kidney, inputting the feature PE3kidney into the fourth CRM to acquire a feature PE4kidney, and inputting the feature PE4kidney into the fifth CRM to acquire a feature PE5kidney;inputting the i-th liver nuclei image Yiliver in the training set into the first CRM to acquire a feature PE1liver, inputting the feature PE1liver into the second CRM to acquire a feature PE2liver, inputting the feature PE2liver into the third CRM to acquire a feature PE3liver, inputting the feature PE3liver into the fourth CRM to acquire a feature PE4liver, and inputting the feature PE4liver into the fifth CRM to acquire a feature PE5liver;inputting the i-th breast nuclei image Yibreast in the training set into the first CRM to acquire a feature PE1breast, inputting the feature PE1breast into the second CRM to acquire a feature PE2breast, inputting the feature PE2breast into the third CRM to acquire a feature PE3breast, inputting the feature PE3breast into the fourth CRM to acquire a feature PE4breast, and inputting the feature PE4breast into the fifth CRM to acquire a feature PE5breast;inputting the i-th colon nuclei image Yicolon in the training set into the first CRM to acquire a feature PE1colon, inputting the feature PE1colon into the second CRM to acquire a feature PE2colon, inputting the feature PE2colon into the third CRM to acquire a feature PE3colon inputting the feature PE3colon into the fourth CRM to acquire a feature PE4colon, and inputting the feature PE4colon into the fifth CRM to acquire a feature PE5colon; andinputting the i-th stomach nuclei image Yistomach in the training set into the first CRM to acquire a feature PE1stomach, inputting the feature PE1stomach into the second CRM to acquire a feature PE2stomach, inputting the feature PE2stomach into the third CRM to acquire a feature PE3stomach, inputting the feature PE3stomach into the fourth CRM to acquire a feature PE4stomach; and inputting the feature PE4stomach into the fifth CRM to acquire a feature PE5stomach;g-3) constructing the image decoder of the image module, comprising a first GRU module, a second GRU module, a third GRU module, and a fourth GRU module;constructing the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module, each comprising a first convolutional layer, a first ReLU activation function, a second convolutional layer, a second ReLU activation function, and an upsampling layer in sequence;inputting the feature PE5brain into the first CRM to acquire a feature PO1brain, inputting the feature PO1brain into the second CRM to acquire a feature PO2brain, inputting the feature PO2brain into the third CRM to acquire a feature PO3brain, and inputting the feature PO3brain into the fourth CRM to acquire a brain nuclei segmentation result image PO4brain;inputting the feature PE5kidney into the first CRM to acquire a feature PO1kidney, inputting the feature PO1kidney into the second CRM to acquire a feature PO2kidney, inputting the feature PO2kidney into the third CRM to acquire a feature PO3kidney, and inputting the feature PO3kidney into the fourth CRM to acquire a kidney nuclei segmentation result image PO4kidney;inputting the feature PE5liver into the first CRM to acquire a feature PO1liver, inputting the feature PO1liver into the second CRM to acquire a feature PO2liver inputting the feature PO2liver into the third CRM to acquire a feature PO3liver, and inputting the feature PO3liver into the fourth CRM to acquire a liver nuclei segmentation result image PO4liver;inputting the feature PE5breast into the first CRM to acquire a feature PO1breast, inputting the feature PO1breast into the second CRM to acquire a feature PO2breast, inputting the feature PO2breast into the third CRM to acquire a feature PO3breast, and inputting the feature PO3breast into the fourth CRM to acquire a breast nuclei segmentation result image PO4breast;inputting the feature PE5colon into the first CRM to acquire a feature PO1colon, inputting the feature PO1colon into the second CRM to acquire a feature PO2colon, inputting the feature PO2colon into the third CRM to acquire a feature PO3colon, and inputting the feature PO3colon into the fourth CRM to acquire a colon nuclei segmentation result image PO4colon; andinputting the feature PE5stomach into the first CRM to acquire a feature PO1stomach, inputting the feature PO1stomach into the second CRM to acquire a feature PO2stomach, inputting the feature PO2stomach into the third CRM to acquire a feature PO3stomach, and inputting the feature PO3stomach into the fourth CRM to acquire a stomach nuclei segmentation result image PO4stomach; andg-4) constructing the GAP module of the image module, comprising a batch normalization (BN) layer, a ReLU activation function, and an adaptive average pooling layer;inputting the feature PE5brain into the GAP module to acquire a feature PGbrain inputting the feature PE5kidney into the GAP module to acquire a feature PGkidney, inputting the feature PE5liver into the GAP module to acquire a feature PGliver, inputting the feature PE5breast into the GAP module to acquire a feature PGbreast, inputting the feature PE5colon into the GAP module to acquire a feature PGcolon, and inputting the feature PE5stomach into the GAP module to acquire a feature PGstomach; andconcatenating the feature PGbrain and the text vector N′brain to acquire a feature vector Nmerbrain, concatenating the feature PGkidney and the text vector N′kidney to acquire a feature vector Nmerkidney, concatenating the feature PGliver and the text vector N′liver to acquire a feature vector Nmerliver, concatenating the feature PGbreast and the text vector N′breast to acquire a feature vector Nmerbreast, concatenating the feature PG and the text vector N′colon to acquire a feature vector Nmercolon, and concatenating the feature PGstomach and the text vector N′stomach to acquire a feature vector Nmerstomach.

7. The multi-organ nuclei segmentation method based on the prompt learning according to claim 6, wherein the step h) comprises: h-1) constructing the MLP module of the segmentation network model, comprising a first convolutional layer, a second convolutional layer, and a third convolutional layer in sequence, wherein the first convolutional layer, the second convolutional layer, and the third convolutional layer each comprise a convolutional kernel with a size of 1*1;h-2) inputting the feature vector Nmerbrain into the MLP module to acquire a feature N1brain, inputting the feature vector Nmerkidney into the MLP module to acquire a feature N1kidney, inputting the feature vector Nmerliver into the MLP module to acquire a feature N1liver, inputting the feature vector Nmerbreast into the MLP module to acquire a feature N1breast, inputting the feature vector Nmercolon into the MLP module to acquire a feature N1colon, and inputting the feature vector Nmerstomach into the MLP module to acquire a feature N1stomach; andh-3) inputting the feature N1brain into a Sigmoid function to acquire a parameter θ1brain, inputting the feature N1kidney into the Sigmoid function to acquire a parameter θ1kidney, inputting the feature N1liver into the Sigmoid function to acquire a parameter θ1liver, inputting the feature N1breast into the Sigmoid function to acquire a parameter θ1breast, inputting the feature N1colon into the Sigmoid function to acquire a parameter θ1colon, and inputting the feature N1stomach into the Sigmoid function to acquire a parameter θ1stomach.

8. The multi-organ nuclei segmentation method based on the prompt learning according to claim 7, wherein the step i) comprises: i-1) performing, by a reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1brain, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a first update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1brain, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a first update of the image decoder;i-2) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1kidney, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a second update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1kidney, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a second update of the image decoder;i-3) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1liver, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a third update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1liver, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a third update of the image decoder;i-4) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1breast, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a fourth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1breast, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a fourth update of the image decoder;i-5) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1colon, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a fifth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1colon, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a fifth update of the image decoder; andi-6) performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1stomach, in the first convolutional layer and the second convolutional layer of each of the first CRM, the second CRM, the third CRM, the fourth CRM, and the fifth CRM of the image encoder; performing a convolution, and an operation by the ReLU activation function in sequence to complete a sixth update of the image encoder; performing, by the reshape function in the PyTorch library of Python, a reshape operation, through the parameter θ1stomach, in the first convolutional layer and the second convolutional layer of each of the first GRU module, the second GRU module, the third GRU module, and the fourth GRU module of the image decoder; and performing a convolution, and an operation by the ReLU activation function in sequence to complete a sixth update of the image decoder, thereby acquiring the updated segmentation network model.

9. The multi-organ nuclei segmentation method based on the prompt learning according to claim 1, wherein the step j) comprises: training, by an adaptive moment estimation (Adam) optimizer, the updated segmentation network model through a Dice similarity coefficient (DSC) loss function to acquire the optimized segmentation network model.

10. The multi-organ nuclei segmentation method based on the prompt learning according to claim 6, wherein the step k) comprises: k-1) inputting the i-th brain nuclei image Yibrain in in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final brain nuclei segmentation result image PO′4brain;k-2) inputting the i-th kidney nuclei image Yikidney in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final kidney nuclei segmentation result image PO′4kidney;k-3) inputting the i-th liver nuclei image Yiliver in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final liver nuclei segmentation result image PO′4liver;k-4) inputting the i-th breast nuclei image Yibreast in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final breast nuclei segmentation result image PO′4breast;k-5) inputting the i-th colon nuclei image Yicolon in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final colon nuclei segmentation result image PO′4colon; andk-6) inputting the i-th stomach nuclei image Yistomach in the training set into the image encoder and the image decoder of the image module in the optimized segmentation network model in sequence to acquire a final stomach nuclei segmentation result image PO′4stomach.