Patent Application
20240420326
The present disclosure relates to systems, methods and computer-accessible medium for analyzing information associated with cancer, and more particularly to systems, methods and computer-accessible medium for predicting cancer outcome(s) and/or treatment response(s).
Breast magnetic resonance imaging (MRI) is a highly sensitive modality for detecting breast cancer with a reported sensitivity of more than 80%. Use in screening is often limited to high-risk patients. Diagnostic MRI is also useful for additional indications such as problem solving and patients with recently diagnosed breast cancer.
Thus, it may be beneficial to provide exemplary systems, methods and computer-accessible mediums, which can determine and/or learn to predict risk of cancer recurrence, diagnose cancer or cancer subtype, and/or predict treatment response(s).
According some exemplary embodiments of the present disclosure, a system can be provided which can comprise, e.g., (a) a first feature extractor that extracts at least one feature from a medical image of a tissue to provide an extracted image feature, (b) a second feature extractor that extracts at least one feature from digitized histopathology data of the tissue to provide an extracted histopathology feature. and (c) a processing unit. For example, the processing unit can: (1) processes the extracted image feature and the extracted histopathology feature to provide a multi-modal representation of the tissue, and (2) processes the multimodal representation of the tissue with a procedure to provide a prediction regarding a medical outcome of the tissue.
In additional exemplary embodiments of the present disclosure, a method and/or a computer program product (comprising a non-transitory, computer-readable medium having a computer-readable program encoded therein, the computer readable program adapted to be executed to implement a method) which can e.g., utilize a medical image analysis system, which can comprises: (1) an image receiving portal, (2) a first feature extractor, (3) a histopathology data receiving portal, (4) a second feature extractor, and (5) an output function.
For example, the method and the computer program product can be used to receive—by the image receiving portal—a medical image of a tissue from an image input source. It is also possible to extract—by the first feature extractor—at least one feature from the medical image of the tissue to provide an extracted image feature. The histopathology data receiving portal can be used to receive digitized histopathology data of the tissue from a histopathology data source. The second feature extractor can be used to extract at least one feature from the digitized histopathology data of the tissue to provide an extracted histopathology feature. The extracted image feature and the extracted histopathology feature can be processed to provide a multi-modal representation of the tissue. The multimodal representation of the tissue can be processed with a particular procedure to provide a prediction regarding a medical outcome of the tissue. Further, the output function can be used to output the prediction regarding the medical outcome of the tissue.
These and other objects, features and advantages of the exemplary embodiments of the present disclosure will become apparent upon reading the following detailed description of the exemplary embodiments of the present disclosure, when taken in conjunction with the appended claims.
Further objects, features and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying Figures showing illustrative embodiments of the present disclosure, in which:
Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the Figures, it is done so in connection with the illustrative embodiments and is not limited by the particular embodiments illustrated in the Figures and the appended claims.
The following description of exemplary embodiments provides non-limiting representative examples referencing numerals to particularly describe features and teachings of different aspects of the present disclosure. The exemplary embodiments described should be recognized as capable of implementation separately, or in combination, with other exemplary embodiments from the description of the exemplary embodiments. A person of ordinary skill in the art reviewing the description of the exemplary embodiments should be able to learn and understand the different described aspects of the present disclosure. The description of the exemplary embodiments should facilitate understanding of the exemplary embodiments of the present disclosure to such an extent that other implementations, not specifically covered but within the knowledge of a person of skill in the art having read the description of embodiments, would be understood to be consistent with an application of the exemplary embodiments of the present disclosure.
Exemplary methods, systems, methods, computer programs, kits, devices, and computer-executable code for predicting cancer outcomes and treatment response according to the exemplary embodiments of the present disclosure are described herein. In some exemplary embodiments, the exemplary cancer outcomes can comprise cancer diagnosis, cancer staging, cancer recurrence, response to treatment, treatment benefit, and prognosis.
According to certain exemplary embodiments of the present disclosure, systems, methods and computer-accessible medium can be provided for diagnosing and/or predicting a condition or disease in a subject. Exemplary systems, methods and computer-accessible medium can also be provided for predicting treatment response of a condition and/or a disease in a subject. Further exemplary systems, methods and computer-accessible medium according to the present disclosure can be provided for predicting recurrence of a condition and/or a disease in a subject.
In some exemplary embodiments of the present disclosure, the condition or disease can be cancer. In For example, the cancer can be a solid tumor, a hematological cancer, a metastatic cancer, a soft tissue tumor, or a combination thereof. Additionally, or alternatively, the cancer can be the solid tumor, and wherein the solid tumor is selected from the group consisting of melanoma, pancreatic cancer, breast cancer, colorectal cancer, lung cancer, skin cancer, ovarian cancer, liver cancer, and a combination thereof. Further or in addition, the cancer can be the hematological cancer, and the hematological cancer can be selected from the group consisting of Hodgkin's lymphoma, Non-Hodgkin's lymphoma, acute myeloid leukemia (AML), chronic myeloid leukemia, myelodysplastic syndrome, multiple myeloma, T-cell lymphoma, acute lymphocytic leukemia, and a combination thereof. In some exemplary embodiments of the present disclosure, the Non-Hodgkin's lymphoma can be selected from the group consisting of B cell lymphoma, diffuse large B cell lymphoma (DLBCL), follicular lymphoma, chronic lymphocytic leukemia (B-CLL), mantle cell lymphoma, marginal zone B-cell lymphoma, Burkitt lymphoma, lymphoplasmacytic lymphoma, hairy cell leukemia, and a combination thereof. For example, the T-cell lymphoma can be peripheral T-cell lymphoma.
According to further exemplary embodiments of the present disclosure, the cancer is breast cancer. In some exemplary embodiments of the present disclosure, the breast cancer can be a subtype of breast cancer. Additionally, or alternatively, the breast cancer can be a luminal breast cancer. Luminal breast cancers can include luminal A, luminal B, or luminal C subtypes. In some exemplary embodiments of the present disclosure, the cancer is a nonluminal breast cancer. Nonluminal breast cancer can include triple negative breast cancer, HER2 enriched breast cancer, or nonluminal unknown breast cancer. The subtype of the breast cancer may be unknown.
In some exemplary embodiments of the present disclosure, the breast cancer can be classified based on histological type. Breast cancer histological types can include ductal carcinoma in situ, invasive ductal carcinoma, metastatic carcinoma, adenocarcinoma, invasive lobular carcinoma, invasive mammary carcinoma, papillary carcinoma, or lymphoma. For example, the histological type of the breast cancer can be unknown.
Non-limiting examples of cancers that can be predicted with exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can include cancer cells from the bladder, blood, bone, bone marrow, brain, breast, colon, esophagus, gastrointestine, gum, head, kidney, liver, lung, nasopharynx, neck, ovary, prostate, skin, stomach, pancreas, prostate testis, tongue, cervix, or uterus. Non-limiting examples of cancer histological types can include neoplasm, malignant; carcinoma; carcinoma, undifferentiated; giant and spindle cell carcinoma; small cell carcinoma; papillary carcinoma; squamous cell carcinoma; lymphoepithelial carcinoma; basal cell carcinoma; pilomatrix carcinoma; transitional cell carcinoma; papillary transitional cell carcinoma; adenocarcinoma; gastrinoma, malignant; cholangiocarcinoma; hepatocellular carcinoma; combined hepatocellular carcinoma and cholangiocarcinoma; trabecular adenocarcinoma; adenoid cystic carcinoma; adenocarcinoma in adenomatous polyp; adenocarcinoma, familial polyposis coli; solid carcinoma; carcinoid tumor, malignant; branchiolo-alveolar adenocarcinoma; papillary adenocarcinoma; chromophobe carcinoma; acidophil carcinoma; oxyphilic adenocarcinoma; basophil carcinoma; clear cell adenocarcinoma; granular cell carcinoma; follicular adenocarcinoma; papillary and follicular adenocarcinoma; nonencapsulating sclerosing carcinoma; adrenal cortical carcinoma; endometroid carcinoma; skin appendage carcinoma; apocrine adenocarcinoma; sebaceous adenocarcinoma; ceruminous adenocarcinoma; mucoepidermoid carcinoma; cystadenocarcinoma; papillary cystadenocarcinoma; papillary serous cystadenocarcinoma; mucinous cystadenocarcinoma; mucinous adenocarcinoma; signet ring cell carcinoma; infiltrating duct carcinoma; medullary carcinoma; lobular carcinoma; inflammatory carcinoma; paget's disease, mammary; acinar cell carcinoma; adenosquamous carcinoma; adenocarcinoma w/squamous metaplasia; thymoma, malignant; ovarian stromal tumor, malignant; thecoma, malignant; granulosa cell tumor, malignant; androblastoma, malignant; sertoli cell carcinoma; leydig cell tumor, malignant; lipid cell tumor, malignant; paraganglioma, malignant; extra-mammary paraganglioma, malignant; pheochromocytoma; glomangiosarcoma; malignant melanoma; amelanotic melanoma; superficial spreading melanoma; malig melanoma in giant pigmented nevus; epithelioid cell melanoma; blue nevus, malignant; sarcoma; fibrosarcoma; fibrous histiocytoma, malignant; myxosarcoma; liposarcoma; leiomyosarcoma; rhabdomyosarcoma; embryonal rhabdomyosarcoma; alveolar rhabdomyosarcoma; stromal sarcoma; mixed tumor, malignant; mullerian mixed tumor; nephroblastoma; hepatoblastoma; carcinosarcoma; mesenchymoma, malignant; brenner tumor, malignant; phyllodes tumor, malignant; synovial sarcoma; mesothelioma, malignant; dysgerminoma; embryonal carcinoma; teratoma, malignant; struma ovarii, malignant; choriocarcinoma; mesonephroma, malignant; hemangiosarcoma; hemangioendothelioma, malignant; kaposi's sarcoma; hemangiopericytoma, malignant; lymphangiosarcoma; osteosarcoma; juxtacortical osteosarcoma; chondrosarcoma; chondroblastoma, malignant; mesenchymal chondrosarcoma; giant cell tumor of bone; ewing's sarcoma; odontogenic tumor, malignant; ameloblastic odontosarcoma; ameloblastoma, malignant; ameloblastic fibrosarcoma; pinealoma, malignant; chordoma; glioma, malignant; ependymoma; astrocytoma; protoplasmic astrocytoma; fibrillary astrocytoma; astroblastoma; glioblastoma; oligodendroglioma; oligodendroblastoma; primitive neuroectodermal; cerebellar sarcoma; ganglioneuroblastoma; neuroblastoma; retinoblastoma; olfactory neurogenic tumor; meningioma, malignant; neurofibrosarcoma; neurilemmoma, malignant; granular cell tumor, malignant; malignant lymphoma; hodgkin's disease; hodgkin's; paragranuloma; malignant lymphoma, small lymphocytic; malignant lymphoma, large cell, diffuse; malignant lymphoma, follicular; mycosis fungoides; other specified non-hodgkin's lymphomas; malignant histiocytosis; multiple myeloma; mast cell sarcoma; immunoproliferative small intestinal disease; leukemia; lymphoid leukemia; plasma cell leukemia; erythroleukemia; lymphosarcoma cell leukemia; myeloid leukemia; basophilic leukemia; eosinophilic leukemia; monocytic leukemia; mast cell leukemia; megakaryoblastic leukemia; myeloid sarcoma; and hairy cell leukemia. In some exemplary embodiments, the tumor may comprise an osteosarcoma, angiosarcoma, rhabdosarcoma, leiomyosarcoma, Ewing sarcoma, glioblastoma, neuroblastoma, or leukemia.
In some exemplary embodiments of the present disclosure, the cancer can be characterized by a cancer antigen present on the cancer. For example, the cancer antigen can be a tumor antigen, a stromal antigen, or a hematological antigen. In some exemplary embodiments, the cancer antigen is selected from the group consisting of BCMA, CD19, CD20, CD22, CD30, CD33, FcRH5, PDL1, CD47, CD117 (c-kit), gangloside 2 (GD2), prostate stem cell antigen (PSCA), prostate specific membrane antigen (PMSA), prostate-specific antigen (PSA), carcinoembryonic antigen (CEA), Ron Kinase, c-Met, Immature laminin receptor, TAG-72, BING-4, Calcium-activated chloride channel 2, calcitonin, Cyclin-B1, Cyclin D1, DCP, 9D7, Ep-CAM, EphA3, Gastrin, HE4, Her2/neu, Telomerase, SAP-1, Survivin, NY-ESO-1/LAGE-1, PRAME, SSX-2, Melan-A/MART-1, 5-HIAA, Gp100/pmel17, Tyrosinase, TRP-1/-2, MC1R, β-catenin, BRCA1/2, CDK4, CML66, Fibronectin, p53, Ras, TGF-B receptor, AFP, ETA, MAGE, MUC-1, CA15-3, CA27.29, CA19-9, CA-125, BAGE, GAGE, NY-ESO-1, β-catenin, CDK4, CDC27, a actinin-4, TRP1/gp75, TRP2, gp100, lactate dehydrogenase, Melan-A/MARTI, gangliosides, WT1, EphA3, Epidermal growth factor receptor (EGFR), MART-2, MART-1, MPO, MUC1, MUC2, MUM1, MUM2, MUM3, NA88-1, nuclear matrix protein 22, NPM, OA1, OGT, PAP, PD-L1, RCC, RUI1, RUI2, SAGE, TdT, TPMT, TRG, TRP1, TSTA, Folate receptor alpha, L1-CAM, CAIX, gpA33, GD3, GM2, VEGFR, Intergrins, carbohydrates, IGFIR, EPHA3, TRAILR1, TRAILR2, RANKL, FAP, TGF-beta, hyaluronic acid, collagen, tenascin C, and tenascin W.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be provided to analyze on a computer system a feature of a medical image of a tissue and a visual feature of a histopathology sample of the tissue using a specific procedure, where the specific procedure can be or include a machine-learning procedure trained on a set of images of cancerous anatomies.
The systems, methods and computer-accessible medium according to certain exemplary embodiments of the present disclosure can generate and/or analyze the images of cancerous anatomies can be magnetic resonance images. For example, the cancerous anatomies can be cancerous breasts, and/or the images of cancerous anatomies can be magnetic resonance images of cancerous breasts.
With the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, e.g., prior to the analyzing, it is possible to perform an imaging procedure on the tissue to obtain the medical image of the tissue.
The exemplary systems, methods and computer-accessible medium according to the present disclosure can provide and/or utilize an AI system which can implement a machine learning procedure (e.g., a neural network) that can learn to predict risk of cancer recurrence, diagnose cancer or cancer subtype, or predict treatment response. In various exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the treatment can be a therapeutic regimen of an agent that is therapeutically effective for the target cancer, for example, compounds described herein, and others as are appropriate for a patient's particular circumstances. In further exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, it is possible to predict the likelihood that a patient having breast cancer will experience a relapse or recurrence. For example, the computer-implemented system can be an artificial intelligence (AI) system, and/or the exemplary machine-learning procedure can be trained on a set of images of cancerous anatomies.
Using the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, it is possible to:
Using the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the outcome can be a medical outcome.
With the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the medical outcome can be a cancer diagnosis, cancer staging, cancer recurrence, response to treatment, treatment benefit, prognosis, survival rate, or a combination thereof.
Using the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the medical outcome can be a duration of time to breast cancer relapse, survival rate, patient age at the time of surgery, tumor stage, tumor size, tumor location (e.g., unifocal, multifocal), number of positive nodes, or surgery type, status of one or more biomarker(s), tumor grade, or histological type, or a combination thereof. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, duration of time to breast cancer relapse or prognosis can be assessed by, for example, overall survival, invasive disease-free survival (iDFS), distant disease-free survival (dDFS), metastasis-free interval (MFI), or a combination thereof.
With the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the overall survival can be the time from diagnosis of a specific disease, e.g., breast cancer, to death from any cause. Overall survival rate can be to the percentage of subjects who are alive at a certain time after diagnosis, among all subjects diagnosed with a specific disease, e.g., breast cancer.
Using the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the invasive disease-free survival (iDFS) in the context of breast cancer can be the time from diagnosis of breast cancer to occurrence of any of the following: ipsilateral invasive breast cancer recurrence, regional invasive breast cancer recurrence, distant recurrence, death attributable to any cause, contralateral invasive breast cancer, second nonbreast invasive cancer, or a combination thereof. For example, the distant disease-free survival (dDFS) in the context of breast cancer can be the time from diagnosis to relapse at a distant site or death from any cause. In addition or alternatively, metastasis-free interval (MFI) in the context of breast cancer can be the time from the diagnosis of primary nonmetastatic breast cancer to the date of the first distant metastases.
According to the exemplary embodiments of the systems, methods and computer-accessible medium of the present disclosure, the input modality can be data from an imaging tool. Non-limiting examples of imaging tools include ultrasound, magnetic resonance imaging (MRI), functional MRI (fMRI), dynamic contrast-enhanced MRI (DCE-MRI), breast MRI, cardiac MRI, computed tomography (CT), X-ray, mammography, positron emission tomography (PET), single photon emission computed tomography (SPECT), fluoroscopy, or a combination thereof.
In certain exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the breast MRI 101 can be MRI images from cancerous breasts. For example, the breast MRI 101 can be from a breast cancer patient prior to treatment. The breast MRI 101 can contain at least one pre-contrast T1-weighted sequence. The breast MRI 101 can contain at least two post-contrast T1-weighted sequence.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the input modality can be from a section of histology or histopathology specimen. For example, the digital pathology can be digitized histopathology slides of breast cancer specimens.
Histopathology includes the microscopic examination of specimens, e.g., tissues, obtained or otherwise derived from a subject, e.g., a patient, to assess a disease state. Histopathology specimens can result from processing the specimen, e.g., tissue, in a manner that affixes the specimen, or a portion thereof, to a microscope slide. For example, thin sections of a tissue specimen can be obtained using a microtome or any suitable device, and the thin sections can be affixed to a slide. Any suitable specimen affixation methods can be used. Non-limiting examples of affixed specimen include formalin-fixed, paraffin-embedded tissue (FFPE), flash frozen tissue using dry ice or liquid nitrogen, cryopreserved tissue, and zinc-fixed tissue.
In further exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the tissue can be human breast tissue and/or a human body part that can be in need of inspection for possible disease, a human body part that is in need of inspection for possible cancer, a human breast that is in need of inspection for possible breast cancer, or a human breast that is afflicted with breast cancer.
The specimen can be further processed, for example, by applying a stain to assist in visualization. Any suitable stains for visualizing cells and tissues can be used. Non-limiting examples of the stains include Haemotoxylin and Eosin (H&E), methylene blue, Masson's trichome, Congo red, Oil Red O, silver nitrate, melanin, Gomori trichrome, Mallory trichrome, Alcian blue, Crystal violet, toluidine blue, and safranin.
The stain can visualize a specific biomarker(s). A biomarker(s) can be a protein, DNA, or RNA biomarker. Any suitable labeling methods can be used to visualize the biomarkers. Staining techniques that depend on the use of labeled detection reagents that specifically bind to a marker of interest, such as immunofluorescence, immunohistochemistry, in situ hybridization, RNAScope can be used.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the clinical variables can comprise categorical and numerical variables that contain information about the subject, e.g., a patient and the condition or disease. Non-limiting examples of clinical variables can include tumor stage, tumor grade, patient's age at diagnosis, histological subtype, patient's age at the time of surgery, menopausal status, number of positive nodes (N+), number of nodules, surgery type, and molecular biomarker status. Non-limiting examples of biomarkers include estrogen (ER), progesterone (PR), HER-2, BRCA1, BRCA2, TP53, Ki-67, and a combination thereof. The molecular biomarker status can be the presence or absence of one or more mutations (i.e., BRCA1 or BRCA2). The molecular biomarker status can be gene expression level (i.e., mRNA level).
In yet further exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, molecular biomarker status can be determined using qPCR. The molecular biomarker status can be determined using ELISA, Sanger sequencing, microarray, and/or next-generation sequence (NGS). Non-limiting examples of NGS includes whole-genome sequencing (WGS), whole-exome sequencing (WES), RNA sequencing, ATAC sequencing, bisulfite sequencing, and chromatin immunoprecipitation (ChIP) sequencing. Further or alternatively, the molecular biomarker can be an epigenetic marker and/or a plurality of biomarkers.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the clinical variables can be in the form of clinical electronic health record (EHR) data. The EHR data can be stored in a software. Non-limiting examples of EHR software include enterprise EHR software, software as a service (SaaS) EHR, custom EHR builds, on-site EHR data storage, EHR data remotely hosted on dedicated servers, cloud-based EHR data storage, certified EHR, stand-alone EHR, or integrated EHR and EPM systems.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the information from the electronic health record can be a tumor stage of the tissue and/or a status of a molecular biomarker in a subject that provided the tissue.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can comprise 1, 2, 3 or more input modalities, or at most 1, 2, or 3 input modalities. For example, the inputs can be the same or different input modality. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the input modality can be patient outcomes.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can comprise at least one, two, or three feature extractors. Alternatively, the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can comprise at most one, two, or three feature extractors.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the feature extractor can take various forms. Non-limiting examples of a feature extractor include: convolutional neural networks; recurrent neural networks; autoencoders; transformer networks; and ensembles thereof. The feature extractor can be used to extract information from the input modalities, such as from imaging data or digital pathology. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the feature extractor can extract at least one feature from a medical image of a tissue to provide an extracted image feature to provide an extracted histopathology feature. The feature extractor can extract at least one feature from digitized histopathology data of a tissue. For example, as shown in
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the medical image can be a magnetic resonance image of a human body part that is in need of inspection for possible disease, a magnetic resonance image of a human body part that is in need of inspection for possible cancer, a magnetic resonance image of a human breast that is in need of inspection for possible breast cancer, or a magnetic resonance image of a human breast that is afflicted with breast cancer. The extracted image feature can be a visual detail that suggests presence of a cancer.
In certain exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the digitized histopathology data of the tissue can be or include an image of a digitized histopathology slide of the tissue, an image of a digitized histopathology slide of a human breast, an image of a digitized histopathology slide of a human body part that is in need of inspection for possible disease, an image of a digitized histopathology slide of a human body part that is in need of inspection for possible cancer, an image of a digitized histopathology slide of a human breast that is in need of inspection for possible breast cancer, or an image of a digitized histopathology slide of a human breast that is afflicted with breast cancer. For example, the extracted histopathology feature can be a visual detail that suggests presence of a cancer.
After extracting information by the feature extractor 104a and the feature extractor 104b, the machine learning system 100, according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, can generate a data representation. For example, the data representation can be low-dimensional representations of the input modality generated by the feature extractor feature extractor 104a and feature extractor 104b. Further or alternatively, the MRI representation 105 and/or the histopathology representation 106 can be low-dimensional representations generated by the feature extractor 104a and feature extractor 104b, respectively.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the multi-modal representation 107 can be a combined data structure derived from multiple input modalities. The representation can be a mathematical vector, matrix, or tensor. Non-limiting examples of input modalities include: mathematical vectors; histopathology representations; and clinical variables. The multi-modal representation 107 can be a mathematical vector that is a result of concatenating MRI Representation 105, Histopathology Representation, 106 and clinical variables 103. The input modalities can be combined by a combining operation. Non-limiting examples of a combining operation include methods such as: concatenation; element-wise addition; element-wise multiplication; weighted combinations; statistical fusion methods; deep-learning-based multi-modal fusion; transformer-based fusion; autoencoder fusion; and any other machine learning or statistical learning method suitable for combining different representations.
For example, the downstream model 108 can be trained to generate predictions 109 using the multi-modal representation 107. The downstream model 108 can be a trained, machine-learning procedure. The downstream model 108 can use various forms of machine learning approaches. Non-limiting examples of machine learning approaches include gradient boosting, a neural network, and ensembles thereof. Gradient boosting is a machine learning technique used to construct an ensemble of simpler models, such as decision trees, to optimize and improve the overall prediction accuracy. The downstream model 108 can iteratively improve each new model based on the mistakes made by previous ones, thereby reducing the bias and variance of the final model. Non-limiting examples of gradient boosting techniques include Catboost or XGBoost. The neural network described herein can comprise multiple fully connected layers.
Predictions 109 made by the system 100 can be in the form of probabilities comprising floating point values. The floating point values can be in a range between 0 and 1, where O is a least likely probability and 1 is a most likely probability. For example, the floating point value can be about 0, about 0.05, about 0.1, about 0.15, about 0.2, about 0.25, about 0.3, about 0.35, about 0.4, about 0.45, about 0.5, about 0.55, about 0.6, about 0.65, about 0.7, about 0.75, about 0.8, about 0.85, about 0.9, about 0.95, and/or about 1.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the predictions 109 can be probabilities of breast cancer recurrence, e.g., at different time horizons. For example, the time horizons can be in the range of 3-10 years, about 6 months, about 2 years, about 3 years, about 4 years, about 5 years, about 6 years, about 7 years, about 8 years, about 9 years, and/or about 10 years.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the predictions 109 can be predictions of possible therapies on impacting the probability of cancer recurrence,
regarding the medical outcome, e.g., of the tissue can be a likelihood of a relapse of a disease, a likelihood of a relapse of cancer, a likelihood of a relapse of breast cancer, a likelihood of favorable response to a therapy for a condition present in the tissue, a likelihood of favorable response to a therapy for cancer present in the tissue, or a likelihood of favorable response to a therapy for breast cancer present in the tissue.
The exemplary embodiments of systems, methods and computer-accessible medium according to the present disclosure can be provided for training the AI system. For example, the AI system can comprise a machine learning procedure, such as a neural network, and/or can be a trained machine learning procedure. The AI system can be a network.
The exemplary AI system as described herein can be configured to undergo at least one training phase, whereas the machine learning software module can be trained to carry out one or more tasks including data extraction, data analysis, and generation of output, and/or using a two-stage training process, comprising, e.g.:
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, during the pretraining feature extractor stage, the feature extractors can be pre-trained on an initial task to generate meaningful representations of the input data, e.g., MRI, digital pathology, clinical variables, auxiliary clinical variables, or patient outcomes. The task can be supervised or self-supervised.
In various exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, to generate Data Representations from feature extractors, the final layer (i.e., the classification layer) can be removed, and the low-dimensional feature representation in the penultimate layer is saved (Data Representation). Input to the Downstream Model 460 can comprise concatenated low-dimensional representations of MRI, Digital Pathology and Clinical Variables. Downstream Model 460 can use a plurality of input modalities and learn to map the input modalities to target output 465, such as a probability of breast cancer risk recurrence.
According to additional exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, various tasks can be optimized by the Downstream Model 460. The Downstream Model 460 can learn to classify the probability of breast cancer risk recurrence within a pre-specified number of years, and/or to optimize a time-to-event model (i.e., a survival analysis model), such as Cox proportional hazards, AFT (Accelerated Failure Time), or discrete time model.
For example, the Downstream Model 460 can determine that the subject is at risk of a breast cancer recurrence of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more.
The Downstream Model 460 can determine that the subject is at risk of a breast cancer recurrence at an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999%, or more.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the exemplary AI system can be trained to predict treatment benefit or response. For example, the AI system can compare two or more available treatment options for which the subject is eligible. Further or alternatively, the AI system can predict whether a therapy can be successful, how long the therapy can take, or be used to determine whether a new therapy is necessary. For example, the supervising physician can be directed to administer a treatment option based at least in part on the outcome of the treatment response prediction. Non-limiting examples of treatment options include all those discussed herein.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the input Clinical Variables 515, 520 can be manipulated so that the Clinical variables 515, 520 represent different treatment options. For example, treatment option 1, 515 can be tamoxifen and Herceptin, while treatment option 2, 520 can be tamoxifen alone. Multiple modified Inputs can be processed through the AI System, and a Prediction 575, 580 can be made. Then, Predictions, 575, 580 can be compared across various treatment options, revealing which Input modification is most or least likely to yield desired patient outcomes.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the AI system can optimize therapies for the subject. For example, any suitable causal inference techniques can be used to allow the AI system to make causal inferences.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be used by healthcare providers (e.g., primarily medical oncologists, radiation oncologists, or breast surgeons) to make more informed decisions about treatment.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the exemplary AI system can be used to escalate or de-escalate treatment (i.e., chemotherapy, hormone therapy, immunotherapy, biologics, radiation therapy, breast surgery). An exemplary escalation or de-escalation can include adding or removing a drug to/from the systemic treatment regimen; shortening the duration of treatment; changing the type of treatment; or lowering or increasing the dosage of a drug.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the prediction regarding a medical outcome of the tissue can be a likelihood of favorable response to a therapy for cancer present in the tissue. For example, the prediction regarding a medical outcome of the tissue can be a likelihood of favorable response to a therapy for breast cancer present in the tissue.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can also comprise administering to a subject that provided the sample a therapeutic intervention that corresponds to the prediction regarding the medical outcome of the tissue.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the AI system can be used to determine whether a specific drug (such as a chemotherapeutic agent) or type of treatment is suitable for a patient. For example, the AI system can be used to determine patient population with HER2-positive breast cancer that can benefit from 6-months versus 12-months trastuzumab treatment; patient population with HER2-positive breast cancer that can benefit from trastuzumab treatment alone versus trastuzumab together with pertuzumab, and patient population planned for chemotherapy treatment that can benefit from the addition of anthracyclines in the regimen.
For example, the AI System can be used to predict the response of a chemotherapeutic agent. Non-limiting examples of chemotherapeutic agents include: 13-cis-retinoic acid (isotretinoin, ACCUTANE®), 2-CdA (2-chlorodeoxyadenosine, cladribine, LEUSTATIN™), 5-azacitidine (azacitidine, VIDAZA®), 5-fluorouracil (5-FU, fluorouracil, ADRUCIL®), 6-mercaptopurine (6-MP, mercaptopurine, PURINETHOL®), 6-TG (6-thioguanine, thioguanine, THIOGUANINE TABLOID®), abraxane (paclitaxel protein-bound), actinomycin-D (dactinomycin, COSMEGEN®), alitretinoin (PANRETIN®), all-transretinoic acid (ATRA, tretinoin, VESANOID®), altretamine (hexamethylmelamine, HMM, HEXALEN®), amethopterin (methotrexate, methotrexate sodium, MTX, TREXALL™ RHEUMATREX®), amifostine (ETHYOL®), arabinosylcytosine (Ara-C, cytarabine, CYTOSAR-U®), arsenic trioxide (TRISENOX®), asparaginase (Erwinia L-asparaginase, L-asparaginase, ELSPAR®, KIDROLASER), BCNU (carmustine, BiCNU®), bendamustine (TREANDA®), bexarotene (TARGRETIN®), bleomycin (BLENOXANE®), busulfan (BUSULFEX®, MYLERAN®), calcium leucovorin (Citrovorum Factor, folinic acid, leucovorin), camptothecin-11 (CPT-11, irinotecan, CAMPTOSAR®), capecitabine (XELODA®), carboplatin (PARAPLATIN®), carmustine wafer (prolifeprospan 20 with carmustine implant, GLIADEL® wafer), CCI-779 (temsirolimus, TORISEL®), CCNU (lomustine, CeeNU), CDDP (cisplatin, PLATINOL®, PLATINOL-AQ®), chlorambucil (leukeran), cyclophosphamide (CYTOXAN®, NEOSAR®), dacarbazine (DIC, DTIC, imidazole carboxamide, DTIC-DOME®), daunomycin (daunorubicin, daunorubicin hydrochloride, rubidomycin hydrochloride, CERUBIDINE®), decitabine (DACOGEN®), dexrazoxane (ZINECARD®), DHAD (mitoxantrone, NOVANTRONE®), docetaxel (TAXOTERE®), doxorubicin (ADRIAMYCIN®, RUBEX®), epirubicin (ELLENCE™), estramustine (EMCYT®), etoposide (VP-16, etoposide phosphate, TOPOSAR®, VEPESID®, ETOPOPHOS®), floxuridine (FUDR®), fludarabine (FLUDARA®), fluorouracil (cream) (CARAC™, EFUDEX®, FLUOROPLEX®), gemcitabine (GEMZAR®), hydroxyurea (HYDREA®, DROXIA™, MYLOCEL™), idarubicin (IDAMYCIN®), ifosfamide (IFEX®), improsulfan, ixabepilone (IXEMPRA™), LCR (leurocristine, vincristine, VCR, ONCOVIN®, VINCASAR PFS®), L-PAM (L-sarcolysin, melphalan, phenylalanine mustard, ALKERAN®), mechlorethamine (mechlorethamine hydrochloride, mustine, nitrogen mustard, MUSTARGEN®), mesna (MESNEX™), mitomycin (mitomycin-C, MTC, MUTAMYCIN®), nelarabine (ARRANON®), oxaliplatin (ELOXATIN™), paclitaxel (TAXOL®, ONXAL™), pegaspargase (PEG-L-asparaginase, ONCOSPAR®), PEMETREXED (ALIMTA®), pentostatin (NIPENT®), piposulfan, procarbazine (MATULANE®), streptozocin (ZANOSAR®), temozolomide (TEMODAR®), teniposide (VM-26, VUMON®), TESPA (thiophosphoamide, thiotepa, TSPA, THIOPLEX®), topotecan (HYCAMTIN®), vinblastine (vinblastine sulfate, vincaleukoblastine, VLB, ALKABAN-AQ®, VELBAN®), vinorelbine (vinorelbine tartrate, NAVELBINE®), and vorinostat (ZOLINZA®).
The AI System can be used to predict the response of a biologic. Biologics useful in the treatment of cancers and a binding molecule as described herein can be administered, for example, in conjunction with such known biologics. Non-limiting examples for treatment of breast cancer include: HERCEPTIN® (trastuzumab); FASLODEX® (fulvestrant); ARIMIDEX® (anastrozole); Aromasin® (exemestane); FEMARA® (letrozole); and NOLVADEX® (tamoxifen). Other biologics with which the binding molecules as described herein can be combined include: AVASTIN® (bevacizumab); and ZEVALIN® (ibritumomab tiuxetan).
Non-limiting examples of biologics for the treatment of colorectal cancer include: AVASTIN®; ERBITUX® (cetuximab); GLEEVEC® (imatinib mesylate); and ERGAMISOL® (levamisole hydrochloride). Non-limiting examples for the treatment of lung cancer include: TARCEVA® (erlotinib HCL). Non-limiting examples for the treatment of multiple myeloma include: VELCADE® (bortezomib). Additional biologics include THALIDOMID® (thalidomide).
In some exemplary embodiments, the AI System can be used to predict the response of a cancer therapeutic antibody. Non-limiting examples of cancer therapeutic antibodies include: 3F8, abagovomab, adecatumumab, afutuzumab, alacizumab pegol, alemtuzumab (CAMPATH®, MABCAMPATH®), altumomab pentetate (HYBRI-CEAKER®), anatumomab mafenatox, anrukinzumab (IMA-638), apolizumab, arcitumomab (CEA-SCAN®), bavituximab, bectumomab (LYMPHOSCAN®), belimumab (BENLYSTA®, LYMPHOSTAT-B®), besilesomab (SCINTIMUN®), bevacizumab (AVASTIN®), bivatuzumab mertansine, blinatumomab, brentuximab vedotin, cantuzumab mertansine, capromab pendetide (PROSTASCINT®), catumaxomab (REMOVABR), CC49, cetuximab (C225, ERBITUX®), citatuzumab bogatox, cixutumumab, clivatuzumab tetraxetan, conatumumab, dacetuzumab, denosumab (PROLIA®), detumomab, ecromeximab, edrecolomab (PANOREX®), elotuzumab, epitumomab cituxetan, epratuzumab, ertumaxomab (REXOMUN®), etaracizumab, farletuzumab, figitumumab, fresolimumab, galiximab, gemtuzumab ozogamicin (MYLOTARG®), girentuximab, glembatumumab vedotin, ibritumomab (ibritumomab tiuxetan, ZEVALIN®), igovomab (INDIMACIS-125®), intetumumab, inotuzumab ozogamicin, ipilimumab, iratumumab, labetuzumab (CEA-CIDE®), lexatumumab, lintuzumab, lucatumumab, lumiliximab, mapatumumab, matuzumab, milatuzumab, minretumomab, mitumomab, nacolomab tafenatox, naptumomab estafenatox, necitumumab, nimotuzumab (THERACIM®, THERALOC®), nofetumomab merpentan (VERLUMA®), ofatumumab (ARZERRA®), olaratumab, oportuzumab monatox, oregovomab (OVAREX®), panitumumab (VECTIBIX®), pemtumomab (THERAGYN®), pertuzumab (OMNITARG®), pintumomab, pritumumab, ramucirumab, ranibizumab (LUCENTIS®), rilotumumab, rituximab (MABTHERA®, RITUXAN®), robatumumab, satumomab pendetide, sibrotuzumab, siltuximab, sontuzumab, tacatuzumab tetraxetan (AFP-CIDER), taplitumomab paptox, tenatumomab, TGN1412, ticilimumab (tremelimumab), tigatuzumab, TNX-650, tositumomab (BEXXAR®), trastuzumab (HERCEPTIN®), tremelimumab, tucotuzumab celmoleukin, veltuzumab, volociximab, votumumab (HUMASPECT®), zalutumumab (HUMAX-EGFR®), and zanolimumab (HUMAX-CD4®).
Additional non-limiting examples of cancer therapeutic agents include: alkylating agents (e.g., thiotepa and cyclosphosphamide), alkyl sulfonates (e.g., busulfan, improsulfan, and piposulfan), aziridines (e.g., such as benzodopa, carboquone, meturedopa, and uredopa), ethylenimines and methylamelamines (e.g., altretamine, triethylenemelamine, trietylenephosphoramide, triethiylenethiophosphoramide, and trimethylolomelamine), acetogenins (e.g., bullatacin and bullatacinone), camptothecin (e.g., topotecan); bryostatin; callystatin, CC-1065 (e.g., adozelesin, carzelesin and bizelesin), cryptophycins (e.g., cryptophycin 1 and cryptophycin 8), dolastatin, duocarmycin (e.g., KW-2189 and CB1-TM1); eleutherobin, pancratistatin, sarcodictyin, spongistatin, nitrogen mustards (e.g., chlorambucil, chlornaphazine, cholophosphamide, estramustine, ifosfamide, mechlorethamine, mechlorethamine oxide hydrochloride, melphalan, novembichin, phenesterine, prednimustine, trofosfamide, and uracil mustard) nitrosureas (e.g., carmustine, chlorozotocin, fotemustine, lomustine, nimustine, and ranimnustine), aclacinomysins, actinomycin, authrarnycin, azaserine, bleomycins, cactinomycin, carabicin, carminomycin, carzinophilin, chromomycinis, dactinomycin, daunorubicin, detorubicin, 6-diazo-5-oxo-L-norleucine, doxorubicin (e.g., morpholino-doxorubicin, cyanomorpholino-doxorubicin, 2-pyrrolino-doxorubicin and deoxydoxorubicin), epirubicin, esorubicin, idarubicin, marcellomycin, mitomycins, such as mitomycin C, mycophenolic acid, nogalarnycin, olivomycins, peplomycin, potfiromycin, puromycin, quelamycin, rodorubicin, streptonigrin, streptozocin, tubercidin, ubenimex, zinostatin, and zorubicin, anti-metabolites (e.g., methotrexate and 5-fluorouracil (5-FU)), folic acid analogues (e.g., denopterin, pteropterin, and trimetrexate), purine analogs (e.g., fludarabine, 6-mercaptopurine, thiamiprine, and thioguanine) pyrimidine analogs (e.g., ancitabine, azacitidine, 6-azauridine, carmofur, cytarabine, dideoxyuridine, doxifluridine, enocitabine, and floxuridine); androgens (e.g., calusterone, dromostanolone propionate, epitiostanol, mepitiostane, and testolactone), anti-adrenals (e.g., as mitotane and trilostane), folic acid replenisher (e.g., frolinic acid; aceglatone; aldophosphamide glycoside), aminolevulinic acid, eniluracil, amsacrine, bestrabucil, bisantrene, edatrexate, defofamine, demecolcine, diaziquone, elformithine, elliptinium acetate, epothilone, etoglucid, gallium nitrate, hydroxyurea, lentinan, lonidainine, maytansinoids (e.g., such as maytansine and ansamitocins), mitoguazone, mitoxantrone, mopidanmol, nitraerine, pentostatin, phenamet, pirarubicin, losoxantrone, podophyllinic acid, 2-ethylhydrazide, procarbazine, PSKpolysaccharide complex, razoxane, rhizoxin, sizofiran, spirogermanium, tenuazonic acid, triaziquone, 2,2′, 2″-trichlorotriethylamine, trichothecenes (e.g., T-2 toxin, verracurin A, roridin A and anguidine), urethan, vindesine, dacarbazine, mannomustine, mitobronitol, mitolactol, pipobroman, gacytosine, arabinoside (e.g., Ara-C), cyclophosphamide; toxoids (e.g., paclitaxel and docetaxel), gemcitabine, 6-thioguanine, mercaptopurine, platinum coordination complexes (cisplatin, oxaliplatin, and carboplatin), vinblastine, platinum, etoposide (e.g., VP-16), ifosfamide, mitoxantrone, vincristine, vinorelbine, novantrone, teniposide, edatrexate, daunomycin, aminopterin, xeloda, ibandronate, irinotecan (e.g., CPT-11), topoisomerase inhibitor, difluorometlhylornithine (DMFO), retinoids (retinoic acid), capecitabine, carboplatin, procarbazine, and plicamycin.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the AI System can determine that the treatment response at an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999%, or more.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the exemplary AI System described herein can be implemented using a Web Application interface.
For example, the exemplary AI System described herein can be implemented with the Web Application and a hospital system.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can provide AI systems comprising a machine learning procedure (e.g., a neural network) that can learn to predict risk of cancer recurrence, diagnose cancer or cancer subtype, or predict treatment response.
The exemplary AI system of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can use a feature extractor to extract information from any one of the input modalities, such as MRI, imaging data, Digital Pathology, Clinical Variables, Auxiliary Clinical Variables, Patient Outcomes, or a combination thereof. The exemplary feature extractor can be trained using data from any one or a plurality of the input modalities.
A machine learning procedure utilized and/or included in the systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can be configured to undergo at least one training phase wherein the machine learning software module can be trained to carry out one or more tasks including data extraction, data analysis, and generation of output.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the machine learning procedure can be trained using a data set and a target in a manner of supervised learning. the exemplary data set can be divided into a training set, a test set, and, in some exemplary embodiments, a validation set. A target can be specified that contains the correct classification of each input value in the data set. For example, data from an input modality can be repeatedly presented to the machine learning procedure, and for each sample presented during training, the output generated by the machine learning procedure can be compared with the desired target. The difference between the target and the set of input samples can be calculated, and the machine learning procedure can be modified to cause the output to more closely approximate the desired target value. A back-propagation procedure can be utilized to cause the output to more closely approximate the desired target value. After several training iterations, the machine learning procedure output can closely match the desired target for each sample in the input training set. Subsequently, when new input data, not used during training, is presented to the machine learning procedure, the specific procedure can generate an output classification value indicating into which of the categories the new sample is most likely to fall. The machine learning procedure can generalize from the training to interpret new, previously unseen input samples. This exemplary feature of a machine learning procedure according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, allows classification of almost any input data that has a mathematically formulatable relationship to the category to which the data should be assigned.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the machine learning procedure can utilize an individual learning model. An individual learning model can be based on the machine learning procedure having trained on data from a single individual and thus, the machine learning procedure that utilizes an individual learning model can be configured to be used on a single individual on whose data the module was trained.
In addition or alternatively, the machine learning procedure can utilize a global training model. A global training model can be based on the machine learning procedure having trained on data from multiple individuals and thus, a machine learning procedure that utilizes a global training model can be configured to be used on multiple individuals.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the machine learning procedure can utilize a simulated training model. A simulated training model is based on the machine learning procedure having trained on data from the input modalities.
Unsupervised learning can be used, in exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, to train a machine learning procedure to use input data such as, for example, MRI data and output, for example, a risk of cancer recurrence. Unsupervised learning, in exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, can include a feature extraction, which can be performed by the machine learning module on the input data. Extracted features can be used for visualization, for classification, for subsequent supervised training, and more generally for representing the input for subsequent storage or analysis. For example, each training case can comprise of a plurality of input modalities.
Machine learning procedure that are suitable for unsupervised training include k-means clustering, mixtures of multinomial distributions, affinity propagation, discrete factor analysis, hidden Markov models, Boltzmann machines, restricted Boltzmann machines, autoencoders, convolutional autoencoders, recurrent neural network autoencoders, and long short-term memory autoencoders.
The exemplary machine learning procedure can include a training phase and a prediction phase. The training phase can provide data to train the machine learning procedure. Non-limiting examples of types of data inputted into a machine learning software module for training can include medical image data, clinical data (e.g., from a health record), clinical variables, auxiliary clinical variables, encoded data, encoded features, and metrics derived from input modalities. Data that are inputted into the machine learning procedure can be used, in the exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, to construct a hypothesis function to determine the risk of cancer recurrence.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, a machine learning procedure can be configured to determine whether the outcome of the hypothesis function was achieved and based on that analysis to determine with respect to the data upon which the hypothesis function was constructed. That is, the outcome may tend to either reinforce the hypothesis function with respect to the data upon which the hypothesis functions were constructed or to contradict the hypothesis function with respect to the data upon which the hypothesis function was constructed. In certain exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, depending on how close the outcome tends to be to an outcome determined by the hypothesis function, the machine learning procedure can adopt, adjust, or abandon the hypothesis function with respect to the data upon which the hypothesis function was constructed. As such, the machine learning procedure of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can dynamically learn through the training phase what characteristics of an input (e.g., data) are most predictive in determining whether the features of a subject's recorded input modalities are predictive of a particular outcome.
Following training, the machine learning procedure of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be used to determine, for example, the risk of cancer recurrence.
The prediction phase can use the constructed and optimized hypothesis function from the training phase to predict the risk of cancer recurrence. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, in the prediction phase, the machine learning procedure can be used to analyze data derived from the input modalities independent of any system or device described herein.
The exemplary probability threshold can be used in conjunction with a final probability to determine whether a given recording matches the trained prediction. Alternatively or in addition, the probability threshold can be used to tune the sensitivity of the trained network. For example, the probability threshold can be 1%, 2%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, 95%, 98%, or 99%. The probability threshold can be adjusted if the accuracy, sensitivity, or specificity falls below a predefined adjustment threshold. The adjustment threshold can be used to determine the parameters of the training period. For example, if the accuracy of the probability threshold falls below the adjustment threshold, the system can extend the training period and/or require additional data from the input modalities. Additional data from the input modalities can be included into the training data, and/or can be used to refine the training data set.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can provide various machine learning (ML) techniques. ML can involve identifying and recognizing patterns in existing data to facilitate making predictions for subsequent data. ML can include a ML model for example, a ML procedure. Machine learning, whether analytical or statistical, can provide deductive or abductive inference based on real or simulated data. The ML model can be a trained model. ML techniques can comprise one or more supervised, semi-supervised, self-supervised, or unsupervised ML techniques. For example, a ML model can be a trained model that is trained through supervised learning (e.g., various parameters are determined as weights or scaling factors).
ML can comprise one or more of regression analysis, regularization, classification, dimensionality reduction, ensemble learning, meta learning, association rule learning, cluster analysis, anomaly detection, deep learning, or ultra-deep learning. Non-limiting examples of ML include: k-means, k-means clustering, k-nearest neighbors, learning vector quantization, linear regression, non-linear regression, least squares regression, partial least squares regression, logistic regression, stepwise regression, multivariate adaptive regression splines, ridge regression, principal component regression, least absolute shrinkage and selection operation (LASSO), least angle regression, canonical correlation analysis, factor analysis, independent component analysis, linear discriminant analysis, multidimensional scaling, non-negative matrix factorization, principal components analysis, principal coordinates analysis, projection pursuit, Sammon mapping, t-distributed stochastic neighbor embedding, AdaBoosting, boosting, gradient boosting, bootstrap aggregation, ensemble averaging, decision trees, conditional decision trees, boosted decision trees, gradient boosted decision trees, random forests, stacked generalization, Bayesian networks, Bayesian belief networks, naïve Bayes, Gaussian naïve Bayes, multinomial naïve Bayes, hidden Markov models, hierarchical hidden Markov models, support vector machines, encoders, decoders, auto-encoders, stacked auto-encoders, perceptrons, multi-layer perceptrons, artificial neural networks, feedforward neural networks, convolutional neural networks, recurrent neural networks, long short-term memory, deep belief networks, deep Boltzmann machines, deep convolutional neural networks, deep recurrent neural networks, generative adversarial networks, vision transformers, long short-term memory networks (LSTM), and masked autoencoders.
Training the ML model of can include, in exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, selecting one or more untrained data models to train using a training data set. The selected untrained data models can include any type of untrained ML models for supervised, semi-supervised, self-supervised, or unsupervised machine learning. The selected, untrained data models can be specified based on input (e.g., user input) specifying relevant parameters to use as predicted variables or other variables to use as potential explanatory variables. For example, the selected, untrained data models can be specified to generate an output (e.g., a prediction) based upon the input. Conditions for training the ML model from the selected untrained data models can likewise be selected, such as limits on the ML model complexity or limits on the ML model refinement past a certain point. The ML model can be trained (e.g., via a computer system such as a server) using the training data set. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, a first subset of the training data set can be selected to train the ML model. The selected, untrained data models can then be trained on the first subset of training data set using appropriate ML techniques, based upon the type of ML model selected and any conditions specified for training the ML model. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, due to the processing power requirements of training the ML model, the selected untrained data models can be trained using additional computing resources (e.g., cloud computing resources). Such training can continue, in exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, until at least one aspect of the ML model is validated and meets selection criteria to be used as a predictive model.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, one or more aspects of the ML model can be validated using a second subset of the training data set (e.g., distinct from the first subset of the training data set) to determine accuracy and robustness of the ML model. Such validation can include applying the ML model to the second subset of the training data set to make predictions derived from the second subset of the training data. The ML model can then be evaluated to determine whether performance is sufficient based upon the derived predictions. The sufficiency criteria applied to the ML model can vary depending upon the size of the training data set available for training, the performance of previous iterations of trained models, or user-specified performance requirements. If the ML model does not achieve sufficient performance, additional training can be performed. Additional training can include refinement of the ML model or retraining on a different first subset of the training dataset, after which the new ML model can again be validated and assessed. When the ML model has achieved sufficient performance, in exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the ML can be stored for present or future use. The ML model can be stored as sets of parameter values or weights for analysis of further input (e.g., further relevant parameters to use as further predicted variables, further explanatory variables, further user interaction data, etc.), which can also include analysis logic or indications of model validity in some instances. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, a plurality of ML models can be stored for generating predictions under different sets of input data conditions. For example, the ML model can be stored in a database (e.g., associated with a server).
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more deep-learning techniques. Deep learning is an example of machine learning (ML) that can be based on a set of procedures that model high-level abstractions in data by using multiple processing layers, with complex structures or otherwise, composed of multiple non-linear transformations. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, a drop out method can be used to reduce overfitting. At each training stage, individual nodes can either be dropped out of the net (e.g., ignored) with probability 1-p or kept with probability p, so that a reduced network is left. Incoming and outgoing edges to a dropped-out node can also be removed. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the reduced network can be trained on the data in that stage. The removed nodes can then be reinserted into the network with the original weights.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more decision tree or random forest techniques. A decision tree can be a supervised ML procedure that can be applied to both regression and classification problems. Decision trees can mimic the decision-making process of a human brain. For example, a decision tree can grow from a root (base condition), and when the tree meets a condition (internal node/feature), the tree splits into multiple branches. The end of the branch that does not split anymore is an outcome (leaf). A decision tree can be generated using a training data set according to the following operations: (1) Starting from a root node (the entire dataset), the procedure can split the dataset in two branches using a decision rule or branching criterion, (2) each of the two branches can generate a new child node, (3) for each new child node, the branching process can be repeated until the dataset cannot be split any further, and/or (4) each branching criterion can be chosen to maximize information gain (e.g., a quantification of how much a branching criterion reduces a quantification of how mixed the labels are in the children nodes). The exemplary labels can be the data or the classification that is predicted by the decision tree.
A random forest regression is an extension of the decision tree model that tends to yield more robust predictions by stretching the use of the training data partition. Whereas a decision tree can make a single pass through the data, a random forest regression can bootstrap 50% of the data (e.g., with replacement) and build many trees. Rather than using all explanatory variables as candidates for splitting, a random subset of candidate variables can be used for splitting to produce trees that have different data and different variables. The predictions from the trees, collectively referred to as the forest, are then averaged to produce the final prediction. Many trees (e.g., one hundred trees) can be included in a random forest model, with a number (e.g., 3, 6, 10, etc.) of terms sampled per split, a minimum of number (e.g., 1, 2, 4, 10, etc.) of splits per tree, and a minimum split size (e.g., 16, 32, 64, 128, 256, etc.). Random forests can be trained in a similar way as decision trees. Training a random forest can include the following operations: (1) select randomly k features from the total number of features; (2) create a decision tree from these k features using the same operations as for generating a decision tree; and (3) repeat the previous two operations until a target number of trees is created.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more long short-term memory (LSTM) techniques. LSTM can be an artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM can use feedback connections. The LSTM architecture can provide a short-term memory for a recurrent neural network (RNN). Such RNN can process not only single data points (such as images), but also entire sequences of data (such as speech or video). The connection weights and biases in the RNN can change once per episode of training, analogously to how physiological changes in synaptic strengths store long-term memories. The activation patterns in the network can change once per time-step, analogously to how the moment-to-moment change in electric firing patterns in the brain store short-term memories. The LSTM architecture can provide a short-term memory for a RNN that can last many (e.g., thousands) timesteps.
In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, a LSTM unit can comprise a cell, an input gate, an output gate, and a forget gate. The exemplary cell can remember values over arbitrary time intervals and the input gate, the output gate, and the forget gate can regulate the flow of information into and out of the cell. Forget gates can be used to decide what information to discard from a previous state by assigning a previous state, compared to a current input, a value between 0 and 1 (e.g., a (rounded) value of 1 can mean to keep the information, and a value of 0 means to discard the information). The input gate can decide which pieces of new information to store in the current state, using the same system as the forget gates. The output gate can control which pieces of information in the current state to output (e.g., by assigning a value from 0 to 1 to the information, considering the previous and current states). Selectively outputting relevant information from the current state can allow that the LSTM network maintains useful, long-term dependencies to make predictions, both in current and future time-steps. LSTM networks can be well-suited to classifying, processing, and making predictions based on time series data, since lags of unknown duration between important events in a time series can occur. LSTMs can resolve the vanishing gradient problem that can be encountered when training traditional RNNs. Relative insensitivity to gap length can be an advantage of LSTM over RNNs, hidden Markov models, and other sequence learning methods in numerous applications.
In certain exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, LSTMs can be used with one or more various types of neural networks (e.g., convolutional neural networks (CNNs), deep neural network (DNNs), recurrent neural networks (RNNs), etc.). In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, CNNs, LSTM, and DNNs can be complementary in modeling capabilities and can be combined in a unified architecture. For example, in such unified architecture, CNNs can be well-suited at reducing frequency variations, LSTMs can be well-suited at temporal modeling, and DNNs can be well-suited for mapping features to a more separable space. For example, input features to a ML model using LSTM techniques in the unified architecture can include segment features for each of a plurality of segments. To process the input features for each of the plurality of segments, the segment features for the segment can be processed using one or more CNN layers to generate first features for the segment. The first features can be processed using one or more LSTM layers to generate second features for the segment. The second features can be processed using one or more fully connected neural network layers to generate third features for the segments, where the third features can be used for classification operations.
According to certain exemplary embodiments of the present disclosure, to process the first features using the one or more LSTM layers to generate the second features, the first features can be processed using a linear layer to generate reduced features having a reduced dimension from a dimension of the first features. The reduced features can be processed using the one or more LSTM layers to generate the second features. Short-term features having a first number of contextual frames can be generated based on the input features, where features generated using the one or more CNN layers can include long-term features having a second number of contextual frames that are more than the first number of contextual frames of the short-term features. In exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, the one or more CNN layers, the one or more LSTM layers, and the one or more fully connected neural network layers can be jointly trained to determine trained values of parameters of the one or more CNN layers, the one or more LSTM layers, and the one or more fully connected neural network layers. In some exemplary embodiments, the input features include log-mel features having multiple dimensions. The input features include one or more contextual frames indicating a temporal context of a signal (e.g., input data). Implementations for such unified architecture can leverage complementary advantages associated with each of a CNN, LSTM, and DNN. For example, convolutional layers can reduce spectral variation in input and help the modeling of LSTM layers. Having DNN layers after LSTM layers can help reduce variation in the hidden states of the LSTM layers. Training the unified architecture jointly can provide a better overall performance. Training in the unified architecture can also remove the need to have separate CNN, LSTM and DNN architectures. By adding multi-scale information into the unified architecture, information can be captured at different time scales.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more support vector machine learning techniques. In machine learning, support vector machines (SVMs) can be supervised learning models with associated learning procedures that analyze data for classification and regression analysis. SVMs can be a robust prediction method, being based on statistical learning. SVMs can be well-suited for domains characterized by the existence of large amounts of data, noisy patterns, or the absence of general theories.
SVMs can map input vectors into high dimensional feature space through non-linear mapping function, chosen a priori. In this high dimensional feature space, an optimal separating hyperplane can be constructed. The optimal hyperplane can then be used to determine things such as class separations, regression fit, or accuracy in density estimation. More formally, a SVM constructs a hyperplane or set of hyperplanes in a high or infinite-dimensional space, which can be used for classification, regression, or other tasks like outlier detection.
Support vectors can be defined as the data points that lie closest to the decision surface (or hyperplane). Support vectors can therefore be the data points that are most difficult to classify and can have direct bearing on the optimum location of the decision surface. Given a set of training examples, each marked as belonging to one of two categories, a SVM training procedure can build a model that assigns new examples to one category or the other, making the procedure a non-probabilistic binary linear classifier. SVM can map training examples to points in space to maximize the width of the gap between the two categories. New examples can then be mapped into that same space and predicted to belong to a category based on which side of the gap the examples fall. In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping inputs into high-dimensional feature spaces.
Within a support vector machine, the dimensionally of the feature space can be large. For example, a fourth-degree polynomial mapping function can cause a 200-dimensional input space to be mapped into a 1.6 billionth dimensional feature space. SVMs assist in discovering knowledge from vast amounts of input data.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more gradient boosting techniques. Gradient boosting is a machine learning technique used in regression and classification tasks, among others. Gradient boosting gives a prediction model in the form of an ensemble of weak prediction models, which are typically decision trees. When a decision tree is the weak learner, the resulting procedure is called gradient-boosted trees. A gradient-boosted trees model is built in a stage-wise fashion as in other boosting methods, but generalizes the other methods by allowing optimization of an arbitrary differentiable loss function.
Exemplary k-Nearest Neighbors
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more K-nearest neighbors (KNN) techniques. KNN is a non-parametric classification method. In KNN classification, the output is a class membership. An object is classified by a plurality vote of neighbors, with the object being assigned to the class most common among the k nearest neighbors (k is a positive integer, typically small). If k=1, then the object is assigned to the class of that single nearest neighbor. In KNN regression, the output is the property value for the object. This value is the average of the values of k nearest neighbors. KNN is a type of classification wherein the function is approximated locally, and computation is deferred until function evaluation. Since this exemplary procedure relies on distance for classification, if the features represent different physical units or come in vastly different scales, then normalizing the training data can improve accuracy.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more Monte Carlo techniques. Monte Carlo is a broad class of computational procedures that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can implement one or more one-hot encoding techniques. One-hot encoding can be used to deal with categorical data. For example, a ML model can use input variables that are numeric. The categorical variables can be transformed in a pre-processing part. Categorical data can be either nominal or ordinal. Ordinal data can have a ranked order of values and can therefore be converted to numerical data through ordinal encoding.
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can provide computer systems or processing units that are programmed to implement methods of the disclosure. A computer system or processing unit can be programmed or otherwise configured to, for example, (i) extract features from any one or a plurality of input modalities, (ii) train and test a trained procedure, (iii) generate data representations using the trained procedure, (iv) concatenate data representations into a multi-modal representation, (v) train and test a Downstream Model, and (vi) generate a prediction.
The exemplary processing unit can regulate various aspects of analysis, calculation, and/or generation of the present disclosure. The processing unit can be or include an electronic device of a user or a computer system that is remotely located with respect to the electronic device. The exemplary electronic device can be a mobile electronic device.
The exemplary processing unit can include a central processing unit, which can be a single core or multi-core processor, or a plurality of processors for parallel processing. The processing unit can also include memory or memory location (e.g., random-access memory, read-only memory, flash memory), electronic storage unit (e.g., hard disk), communication interface (e.g., network adapter) for communicating with one or more other systems, and peripheral devices, such as cache, other memory, data storage and/or electronic display adapters. The memory, storage unit, interface, and peripheral devices can be in communication with the CPU through a communication bus, such as a motherboard. The storage unit can be a data storage unit (or data repository) for storing data. The processing unit can be operatively coupled to a computer network with the aid of the communication interface. The network can be the Internet, an internet and/or extranet, or an intranet and/or extranet that is in communication with the Internet.
The network—in some exemplary cases—can be or include a telecommunication or data network. The network can include one or more computer servers, which can provide distributed computing, such as cloud computing. For example, one or more computer servers may enable cloud computing over the network to perform various aspects of analysis, calculation, and generation of the present disclosure, such as, for example, (i) extract features from any one or a plurality of input modalities, (ii) train and test a trained procedure, (iii) generate data representations using the trained procedure, (iv) concatenate data representations into a multi-modal representation, (v) train and test a Downstream Model, and (vi) generate a prediction. Such cloud computing can be provided by cloud computing platforms such as, for example, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform, and IBM cloud. The network, in some cases, with the aid of the processing unit, can implement a peer-to-peer network, which can permit that devices coupled to the processing unit behave as a client or a server.
The CPU can comprise one or more computer processors or one or more graphics processing units (GPUs). The CPU can execute a sequence of machine-readable instructions, which can be embodied in a program or software. The instructions can be stored in a memory location, such as the memory. The instructions can be directed to the CPU, which can subsequently program or otherwise configure the CPU to implement methods of the present disclosure. Examples of operations performed by the CPU can include fetch, decode, execute, and writeback.
The CPU can be part of a circuit, such as an integrated circuit. One or more other components of the processing unit can be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).
A storage unit can be used store files, such as drivers, libraries, and saved programs. The storage unit can store user data, e.g., user preferences and user programs. The processing unit in some cases can include one or more additional data storage units that are external to the computer system, such as located on a remote server that is in communication with the processing unit through an intranet or the Internet.
The exemplary computer system can communicate with one or more remote processing units through the network. For instance, the processing unit can communicate with a remote computer system of a user. Examples of remote computer systems include personal computers (e.g., portable PC), slate or tablet PCs (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iphone, Android-enabled device, Blackberry®), or personal digital assistants.
Exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system, such as, for example, on the memory or electronic storage unit. The machine executable or machine readable code can be provided in the form of software. During use, the code can be executed by the processor. In some cases, the code can be retrieved from the storage unit and stored on the memory for ready access by the processor. In some exemplary embodiments, the electronic storage unit can be precluded, and machine-executable instructions are stored on memory.
The code can be pre-compiled and configured for use with a machine having a processer adapted to execute the code or can be compiled during runtime. The code can be supplied in a programming language that can be selected to enable the code to execute in a pre-compiled or as-compiled fashion.
Aspects of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, such as the computer system or processing unit, can be embodied in programming. Various aspects of the technology can be provided as executable code and/or associated data that is carried on or embodied in a type of machine readable medium. Machine-executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. Storage type media can include any or all the tangible memory of the computers, processors, or associated modules thereof, such as various semiconductor memories, tape drives, and disk drives, which can provide non-transitory storage at any time for the software programming. All or portions of the software can at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, can allow loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of media that can bear the software elements includes optical, electrical, and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links or optical links, also can be considered as media bearing the software.
An exemplary machine readable medium, such as computer-executable code, can take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s), such as can be used to implement databases, etc., Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media can take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media can be involved in carrying one or more sequences of one or more instructions to a processor for execution.
The processing unit described herein can include or be in communication with an electronic display that comprises a user interface (UI) for providing, for example, (i) a visual display indicative of training and testing of a trained procedure, (ii) a visual display of data indicative of a liver disease state of a subject, (iii) a quantitative measure of a liver disease state of a subject, (iv) an identification of a subject as having a liver disease state, or (v) an electronic report indicative of the liver disease state of the subject. Examples of UIs include, without limitation, a graphical user interface (GUI) and web-based user interface.
Methods and systems of the present disclosure can be implemented by way of one or more exemplary procedures. The exemplary procedure can be implemented by way of software upon execution by the central processing unit. The exemplary procedure can, for example, (i) extract features from any one or a plurality of input modalities, (ii) train and test a trained alg procedure, (iii) generate data representations using the trained procedure, (iv) concatenate data representations into a multi-modal representation, (v) train and test a Downstream Model, and (vi) generate a prediction.
indicates data missing or illegible when filed
indicates data missing or illegible when filed
indicates data missing or illegible when filed
The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can include a subject (e.g., a patient) in need of inspection for possible cancer (e.g., breast cancer) going to a medical clinic to receive a cancer assessment. At the clinic, the subject can receive a breast MRI, a histological sampling of the breast tissue, biopsy of the breast tissue, blood draw, or a combination thereof. Clinical variables, such as medical history can be recorded in the form of electronic health record. Routine histopathological fixation and staining of the breast tissue on a slide(s) can be performed. Assessment of cancer biomarkers can be performed from the blood sample or the breast biopsy. The exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can then incorporate the MRI images, the histopathological slides, or clinical variables to arrive at a medical conclusion (e.g., diagnosis, prognosis, treatment plan).
The exemplary feature extractor according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, can utilize a deep learning (DL) system such as that illustrated in
The dataset used for model training and evaluation included 21,537 bilateral DCE-MRI examinations (n=13,463 patients) who underwent a breast MRI between 2008 and 2020. All examinations were performed with either 1.5-T magnet or 3-T magnet MRI scanners (Table 4). Data included patients reporting for high-risk screening, preoperative planning, routine surveillance, follow-up after suspicious findings in previous MRI examinations, and problem solving (workup of equivocal findings reported in mammography or ultrasound). Patients after bilateral mastectomy, patients after neoadjuvant chemotherapy, and patients with MRI performed to assess implant integrity were excluded. T1-weighted fat-saturated precontrast and at least two postcontrast series were required for the imaging exam to be included in the dataset. The entire dataset was initially split into training, validation, and test subsets with a 60:15:25 ratio. Additional filtering and manual evaluation were performed to ensure consistency of the dataset and ground truth labels. In addition to DCE-MRI examination, associated radiology and pathology reports and patient demographic data (Tables 1 and 2) were collected.
The exemplary feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure performance was validated on three international datasets. The first dataset was collected between 2019 and 2021 and contains 394 DCE-MRI examinations. The remaining two datasets contained 922 MRI examinations and 131 examinations from the T dataset. MRI scanner breakdown is described Table 4.
On the J test set, the feature extractor achieved 0.797 AUROC (0.756 to 0.838) and 0.596 AUPRC (0.522 to 0.674). On the D dataset, the feature extractor reached 0.969 AUROC (0.960 to 0.976) and 0.977 AUPRC (0.971 to 0.982), and on the T dataset, the feature extractor reached 0.966 AUROC (0.942 to 0.985) and 0.973 AUPRC (0.954 to 0.988).
In a retrospective reader procedure, five radiologists interpreted 100 cases sampled from the N test set. All radiologists were board-certified and had between 2 and 12 years of experience interpreting breast MRI exams. Readers achieved an average performance of 0.890 AUROC (range: 0.850 to 0.948) and 0.758 AUPRC (range: 0.712 to 0.868). The feature extractor standalone performance on the reader study subset was 0.924 (0.880 to 0.962) AUROC and 0.784 (0.656 to 0.887) AUPRC. The difference between the feature extractor and radiologists was not statistically significant (Obuchowski-Rockette model, 95% CI AUC difference: 0.09, −0.02; P=0.19). In a head-to-head comparison between the feature extractor's and radiologists' AUROC, the feature extractor was significantly better (P<0.05) than two of the readers (#1 and #2). ROC curves for all readers are presented in
To understand the difference in the feature extractor's performance on the N test set and the J test set, an additional reader study on the J test set was performed. The additional reader study included two radiologists (also included in the N reader study) who interpreted 97 cases (including 35 cancer cases) sampled from the J test set. The feature extractor achieved 0.802 (0.712 to 0.881) AUROC, whereas the two radiologists achieved 0.787 (0.708 to 0.859) and 0.849 (0.776 to 0.916), respectively. Detailed results are in Table 6.
The performance of hybrid models was evaluated by averaging radiologists' and the feature extractor's predictions on the N reader study subset. Hybrid predictions were calculated by averaging predictions of POM made by a radiologist and the feature extractor. Hybrid predictions are simulated, as radiologists were not presented with the feature extractor outputs during exam interpretation. On average, an equally weighted hybrid improved the AUROC by 0.05 and AUPRC by 0.07. This effect was observed each time each radiologist's predictions were averaged, and is illustrated in
This weight could be treated as an operating point and be set specifically for different readers, as the optimal operating point (that maximizes the performance) varied between radiologists, as evidenced in
Analyses on a wide range of patient subgroups in the test set were performed, dividing the subgroups with respect to imaging features, cancer subtypes, and other characteristics. For example,
The performance in subgroups with respect to exam indications were compared and no statistically significant differences (P≥0.05) were found. For example, no difference was observed in model performance in women undergoing high-risk screening MRI versus women undergoing a follow-up exam (P=0.4). No difference was observed when comparing performance in screening MRI versus the extent of disease MRI (P=0.22) or screening MRI versus any nonscreening MRI exam (P=0.11). Model performance in all exam indication subgroups is described in Table 8.
No differences were observed in the feature extractor's performance between patients with various histological cancer subtypes, even when comparing more common cancers (for example, invasive ductal carcinoma) with less common malignancies (for example, invasive lobular carcinoma; AAUC: 1.5; two-sided DeLong's test, P=0.15). When considering patient demographics, the results indicated that the feature extractor appeared to be unbiased, even when the subgroup was not as commonly represented in the training set, as in the case of Black women [n=802 patients in the training set; test set AUROC of 0.91 (0.87 to 0.95)] versus white women [n=9819 in the training set; test set AUROC of 0.93 (0.91 to 0.94)]. The AUC difference between those two groups was not statistically significant (P=0.33).
Personalized Management of Patients with BI-RADS 3 and BI-RADS 4 Lesions
To create a diagnostic decision-making application of the feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, using the feature extractor predictions as an aid in downgrading BI-RADS 4 lesions to BI-RADS 3 was used as a test. Definitions of BI-RADS risk assessment categories were used as specified in the American College of Radiology BI-RADS Atlas fifth edition. The analysis was conducted in two ways, both using the full test set and original BI-RADS of the imaging exams as reported initially by radiologists. First, trade-offs between correctly avoided biopsies and missed cancers at various decision thresholds used to binarize probabilities of malignancy was compared directly. The trade-off was an equally weighted comparison between the number of successfully opted out patients (patients who avoided unnecessary biopsy) and missed cancers. In a second approach, the decision curve analysis (DCA) methodology was used to incorporate patients' and clinicians' preferences into decision-making and explored whether using the model could be clinically beneficial or harmful.
According to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, at an operating point that allows for the identification of 5.4% nonmalignant BI-RADS 4 lesions (avoiding a biopsy), no cancers would be missed. At a different operating point at which the system correctly determined 22.9% of BI-RADS 4 lesions to be nonmalignant (avoiding a biopsy), 10 (2.3%) malignant BI-RADS 4 findings would be missed (determined as nonmalignant by the system). For example,
Similarly, BI-RADS 3 lesions could potentially be downgraded to BI-RADS 2, subsequently leading to patients' return to routine screening instead of short-term follow-up MRI after 6 or 12 months. The feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure correctly downgraded 235 BI-RADS 3 lesions (73.2% of all nonmalignant BI-RADS 3 cases) to BI-RADS 2, missing three cancer cases.
The use of the feature extractor in patients with BI-RADS 4 lesions resulted in net reduction in interventions in low decision thresholds. If a decision threshold is at 5%, the approach resulted in a net reduction of 156 breast biopsies per 1000 patients.
An error analysis of the feature extractor's predictions using the N reader study subset was performed. Predictions of malignancy made by the feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure were compared to predictions made by radiologists. Assessment indicated the feature extractor's predictions matched those of the radiologists.
The feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure behaved correctly when predicting examinations with cancer by giving them a high POM (e.g., see
Using the exemplary MRI Data Representations generated by the MRI feature extractor, a Downstream Model to predict 3-year overall breast cancer recurrence was trained according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure. The Downstream Model was a gradient boosting classifier. The Downstream Model of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure consistently yielded an AUCROC (area under the receiver operating characteristic curve) above 0.7. Selected specific patient subgroups of clinical interest was evaluated with the Downstream Model. For example, a subgroup of hormone-positive, HER2-negative patients, who also received the Oncotype DX recurrence prognostic test (based on genomic data) was evaluated. On the N dataset of patients with available Oncotype DX scores (n≈125), the Downstream Model of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure resulted in a 0.186 (from 0.537 to 0.723) improvement in AUCROC. In the group of patients with available Oncotype DX scores who received adjuvant chemotherapy, the Downstream Model had a 0.123 AUCROC improvement (from 0.597 to 0.720) over Oncotype DX. In the group of patients with available Oncotype DX scores who did not receive adjuvant chemotherapy (n=81), the Downstream Model had a 0.202 AUCROC improvement (from 0.535 to 0.737) over Oncotype DX.
Various model architectures and statistical approaches to predicting breast cancer recurrence were tested according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure. Two types of model architectures were tested: a gradient boosting model and neural network. These models were trained with various statistical approaches to predicting cancer recurrence (classifier, accelerated failure time model, discrete time model). Clinical Variables were used as input. Results are shown in Table 11.
Table 1 summarizes the demographic data and imaging characteristics in the datasets. Values are n (%) unless specified otherwise. BI-RADS risk assessment categories, background parenchymal enhancement (BPE), and the amount of fibroglandular tissue are reported according to the American College of Radiology BI-RADS Atlas fifth edition. Breast-level diagnosis statistics are presented in Table 10. Malignant and benign findings are not mutually exclusive. Thus, the total number of examinations labeled as malignant, benign, or negative can be greater than 100%. A negative diagnosis means that no pathology reports were associated with an examination.
The dataset consists of 21,537 DCE-MRI examinations from 13,463 patients who underwent DCE-MRI between 2008 and 2020. The data were randomly split into training, validation, and test sets with 60, 15, and 25% of the data, respectively. This split was made on a patient level, so that data from one patient could be in only one subset.
All datasets used to train and evaluate the feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure contain breast-level labels describing the presence or absence of benign or malignant findings in either the left or right breast. All benign and malignant labels in the dataset were pathology-proven based on specimen analysis from either a breast biopsy or surgery, done by searching and matching all pathology reports dated 120 days before or after the day of examination.
To maximize the accuracy of the truthing and to remove potentially confounding subgroups, additional filtering of the dataset was performed. In the full dataset, imaging exams where a cancer label was not reliably determined or technical issues with data extraction or data consistency were excluded. A set of rules specific to the test set to further remove label noise was added. Patients with a history of bilateral mastectomy (n=165), patients after neoadjuvant chemotherapy (n=105), and patients with breast implants (n=370) were excluded. Cases were reviewed manually where (i) an examination was initially labeled by the feature extractor as negative and, at the same time, was assigned as BI-RADS 1, 2, or 3 by a radiologist; (ii) an examination was labeled as malignant but was assigned as BI-RADS 1 or 2; and (iii) an examination was labeled as malignant but was assigned as BI-RADS 0, 3, or 6. In situation (i), a 1-year negative follow-up requirement was added, meaning that in the year after the MRI exam date, (a) no pathology reports were associated with the patient; (b) at least one breast imaging exam (mammography, MRI) occurred with BI-RADS category 1, 2, or 3; and (c) no breast imaging exams had BI-RADS category 0, 4, 5, or 6. Negative studies without proper follow-up (n=1135) were excluded. In situation (ii), cases (n=10) with clear mistake to label an examination as both malignant and BI-RADS 1 or 2 were excluded. In situation (iii), all cases (n=433) were manually reviewed and verified correctness of the labels. If necessary, examinations had their labels fixed or were excluded (n=19).
Datasets were collected. All external data underwent the same preprocessing pipeline. The sets were resampled, reoriented to the LPS (left-posterior-superior) orientation, and saved in an appropriate file format.
This exemplary dataset includes 394 DCE-MRI examinations. In 145 imaging exams, at least one breast had pathology-proven breast cancer. In the remaining 249 examinations, no pathology-confirmed cancer was found in any breast.
Labeling and anonymization were performed by a board-certified breast radiologist, and the labels were pathology-proven. Ninety-nine percent of examinations were acquired on a 1.5T Siemens MAGNETOM Sola between December 2019 and August 2021. Indications for a scan varied, but the largest group was patients undergoing a problem-solving MRI after ambiguous findings in other modalities.
After obtaining the dataset, all imaging exams were semi-automatically identified pre- and postcontrast sequences and converted to the NIfTI format. Then, a manual visual review of saved images was performed to confirm accuracy of pre-/postcontrast assignment. From the original dataset, five examinations that did not contain fat-saturated images, three unilateral exams, two examinations missing pathology confirmation, one exam that was confirmed to be postneoadjuvant chemotherapy MRI, and one examination that did not have consistent image size in pre- and postcontrast series were excluded.
The dataset contained 922 examinations (n=922 patients) with invasive breast cancer, meaning that for each of these exams, at least one breast had pathology-proven breast cancer. The dataset was accompanied by detection labels, clinical features, and imaging features. Detection labels were available for all imaging exams and were shared in a tabular form describing the coordinates of 3D cuboid bounding boxes. Images in the D dataset were stored in DICOM format and were preprocessed with the same pipeline as the N dataset. Images were resampled and reoriented to the LPS orientation. Pre- and post-contrast sequences, as required by our model, and performed inference to generate predictions were identified.
Bounding box labels were converted to breast-level labels based on the middle point of bounding boxes. For example, if the central point of a bounding box was located on the left anatomical side of the patient, then a label for left breast was generated. The images were annotated by eight fellowship-trained breast radiologists with 1 to 22 years of post-fellowship experience.
Originally, this dataset included 164 examinations from 139 patients and contains images, clinical data, biomedical data, and DICOM Structured Reporting files. The data in its original form were not suitable for the feature extractor's evaluation. Therefore, a pipeline for generating feature extractor-ready T data with labels was developed. After processing, the dataset contained 131 imaging exams.
Let x∈C, Z, X, Y denote an input. Z, X, and Y are the spatial dimensions of an MRI volume, and C channels are different MRI sequences, that is, pre- and postcontrast series. The neural network of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can generate four probability estimates y{circumflex over ( )}_(lb,) y{circumflex over ( )}_(lm,) y{circumflex over ( )}_(rb,) y{circumflex over ( )}_(rm,)∈[0,1] that indicate the predicted probability of the presence of benign and malignant lesions in each of the patient's breast (b and m represent benign and malignant findings, and l and r represent left and right breasts, respectively). Probabilities of benign findings (y{circumflex over ( )}_(lb,) y{circumflex over ( )}_rb) can be used only as a multitask learning regularization method. Predicted probabilities of malignant findings (y{circumflex over ( )}_(lm,) y{circumflex over ( )}_rm) may be evaluated.
The feature extractors of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be deep residual neural networks with 3D convolutions to detect spatiotemporal features. 3D-ResNet18 backbone can be used with a max pooling layer before linear classifier.
Feature extractors according to the exemplary embodiments of the systems, methods and computer-accessible medium of the present disclosure with pretrained backbones can perform better compared to models trained from scratch. Weights can be used from models pretrained on the Kinetics-400 dataset, which is an action recognition in video dataset.
Various geometrical, intensity-based and MRI-specific augmentations were tested. Affine transformation resulted in consistently beneficial results. During training of the feature extractor of exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, images can be randomly (P=0.5) flipped horizontally in the left-right anatomical axis. When the volume is flipped, labels are flipped as well, such that y_lb→y_rb while y_rb→y_lb and y_lm→y_rm while y_rm→y_lm.
The affine augmentations applied were not based on the tensor sizes but rather on real-life dimensions. To convert into anatomical dimensions, affine matrices can be calculated for each study that defined relationships between pixel and real-life sizes. To compute an affine matrix, image spacing, origin, and direction cosine values that were collected from DICOM metadata can be used. The matrices were necessary to resample initially to the same pixel spacing and reorient all images to the LPS orientation, which is a standard orientation in DICOM convention. Pixels in the matrix followed the anatomical order and went from right toward left, from anterior toward posterior, and from inferior toward superior. Resampling can be performed using linear interpolation.
Subtraction images can be used to improve performance. Subtraction images can be generated by a simple matrix subtraction between postcontrast and precontrast volumes, such that X_(substraction i)=X_(post i)−X_pre, where X_(post) is one of i postcontrast volumes and X_pre is the precontrast volume. Lower range values were not clipped to zero.
Most or all feature extractor models according to the exemplary embodiments of the systems, methods and computer-accessible medium of the present disclosure were trained with the Adam optimizer, and top models for the ensemble were selected after hyperparameter tuning with random search. AUROC for malignant labels was the target metric in the hyperparameter search. The following exemplary parameters were tuned:
Exemplary models according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can be trained with mixed precision using NVIDIA Apex open-source library. Network architecture included group normalization. Group normalization with 16 groups can perform well. Neptune.ai and Weights & Biases can be used for tracking, evaluating, and visualizing experimental results.
During test time, all data samples were transformed 10 times (TTA) and averaged inference results from all TTA samples. This approach according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure can improve accuracy and robustness of feature extractor models. The optimal TTA policy according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure was found on a validation set after running a random search for the following TTA hyperparameters: number of TTA rounds ∈[1, 10], affine scaling ∈[10%, 20%], rotation ∈[10°, 20°], and translation ∈[10 mm, 20 mm] using gamma transformations or blurring. The best-performing TTA policy on the validation set was the one that implemented 10 rounds of TTA, random horizontal flips, and affine transformations of 10% scaling factor, 10° rotation, and 10-pixel translation. Differences between various TTA policies were usually indistinguishable. For single models (e.g., not full ensemble), an improvement of ≈0.005 AUCROC and ≈0.01 AUCPR on the validation set can be observed.
A retrospective reader analysis was designed to compare the standalone clinical performance of the model according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure with radiologists. A single-arm design (readers interpreting MRI examinations only) was included in the study. Five board-certified breast radiology attendings were recruited to participate in the study. 100 imaging exams were randomly selected from the primary test set as a reader study set for readers to interpret. The dataset was enriched with malignant and nonmalignant biopsied cases. Specifically, 40 malignant, 40 benign, and 20 negative studies existed in the reader study set. Readers were informed that the population of the study does not represent typical distribution of patients undergoing breast MRI. Readers had no knowledge about the specific split. Readers were also blinded to any confidential information, prior imaging exams, or indications for the examination. Radiologists had access to all available MRI sequences and were not limited to T1-weighted fat-saturated series that were used as inputs for the feature extractor system.
Readers were provided with a workstation preloaded with examinations. All imaging exams used in the reader study were pseudonymized and stored in a server separate from clinical PACS (picture archiving and communication system) servers. Radiologists had access to the workstation and a data collection tool. Before joining the study, recruited readers had to become familiar with study instructions and the viewer used in the study. Radiologists had to provide the following predictions:
Readers were advised to use BI-RADS likelihood of cancer ranges as a guideline when assigning POM values. If a reader believed that the examination was probably benign (BI-RADS 3), then the reader was to assign a POM value in the (0, 2] interval.
To evaluate whether the shift in data distribution across data sets affected the feature extractor's performance to a similar degree as the shift affected radiologists, an additional, smaller reader exemplary analysis was performed. This study was designed in the same manner as the original reader exemplary analysis on the N dataset. The exemplary analysis included two attending breast radiologists, who also participated in the N reader study. The J reader study had a slightly different study enrichment. The J dataset did not distinguish between benign and negative (nonbiopsied) exams. 35 malignant exams and 62 nonmalignant examinations existed in the reader exemplary analysis subset.
To measure model performance according to exemplary embodiments of the systems, methods and computer-accessible medium according to the present disclosure, ROC and PR were used and AUC was calculated for both (AUCROC and AUCPR) using a nonparametric (trapezoidal) method. Sensitivity and specificity were reported. Specific clinical scenarios were evaluated using partial AUC statistic, as implemented in the partial ROC R package. All or most results, where appropriate, were reported with 95% Cis derived using bootstrapping. When evaluating standalone performance versus reader performance, a single-treatment random-reader random-case model was used based on the Obuchowski-Rockette model. This exemplary method accounted for variability both between readers and cases. The null hypothesis for significance testing was that the average breast-level AUC of the feature extractor DL model equaled the average AUC of radiologists. In subgroup analyses, for comparisons of AUCs of two curves, a two-sided DeLong's test was performed. P<0.05 was considered statistically significant.
For DCA, R packages rmda and dcurves were used to generate the curves, calculate net benefit, and avoid net interventions. A range of reasonable threshold probabilities for evaluated subgroups (BI-RADS 4 and 3) was established and a wide range of threshold probabilities on a full population was reported.
The standardized net benefit (sNB; also known as relative utility) for the opt-out policy (here, downgrading BI-RADS 4 to BI-RADS 3) can be defined as
where prevalence is disease prevalence, a is the decision threshold, TNR is a true-negative rate, and FNR is a false-negative rate.
To avoid overestimating the net benefit, the results were boostrapped with N=2000 replicates and reported decision curves with 95% CIs.
For measuring interreader variability, Fleiss' kappa was used, specifically with Randolph's free-marginal modification, for agreement between positive and negative cases. Fleiss' kappa was calculated on exam level and breast level for readers. In addition, ROC curves were generated for readers and hybrids and measured the sample variance in AUC ROC. Details can be found in Table 7 and
As shown in
Further, the exemplary processing arrangement 4002 can be provided with or include an input/output arrangement 4014, which can include, for example a wired network, a wireless network, the internet, an intranet, a data collection probe, a sensor, etc.
According to certain exemplary embodiments of the present disclosure, a method can be provided that comprises:
The method of para. [00218] can be provided, wherein the processing the extracted image feature and the at least one extracted histopathology feature to provide a multi-modal representation of the tissue processes the extracted image feature and the extracted histopathology feature with information from an electronic health record corresponding to the tissue to provide the multi-modal representation of the tissue.
The method of para. [00219] can be provided, wherein the information from the electronic health record is at least one of (i) a tumor stage of the tissue, or (ii) a status of a molecular biomarker in a subject that provided the tissue.
The method of para. [00219] can be provided, wherein the specific procedure is at least one of (i) a machine-learning procedure, (ii) a trained, machine-learning procedure, or (iii) a machine-learning procedure trained on a set of magnetic resonance images of one or more cancerous breasts.
The method of para. [00219] can be provided, wherein the medical image is at least one of a magnetic resonance image of a human breast or a human body part that is (i) targeted for an inspection for a possible disease, a cancer or a breast cancer, or (ii) afflicted with a possible disease, a cancer or a breast cancer.
The method of para. [00219] can be provided, wherein the extracted image feature is a visual detail that suggests a presence of a cancer.
The method of para. [00219] can be provided, wherein the digitized histopathology data of the tissue is an image of a digitized histopathology slide of at least one of (i) the tissue, (ii) a human breast, (iii) a human body part that is targeted for an inspection for a possible disease, a possible cancer or a possible breast cancer, or that is afflicted with a breast cancer.
The method of para. [00219] can be provided, wherein the extracted histopathology feature is a visual detail that suggests a presence of a cancer.
The method of para. [00219] can be provided, wherein the tissue is a human breast tissue.
The method of para. [00219] can be provided, wherein the tissue is a human body part that is targeted for an inspection for at least one of (i) a possible disease, or (ii) a possible breast cancer.
The method of para. [00219] can be provided, wherein the tissue is a human breast that is afflicted with a breast cancer.
The method of para. [00219] can be provided, wherein the prediction regarding the medical outcome of the tissue is a likelihood of (i) a relapse of a disease or breast cancer, (ii) a favorable response to a therapy for a condition, cancer, or breast cancer present in the tissue.
The method of para. [00219] can be provided, further comprising administering to a subject that provided the sample a therapeutic intervention that corresponds to the prediction regarding the medical outcome of the tissue.
The method of para. [00219] can be provided, wherein the tissue is of a subject, further comprising administering to the subject a therapeutically-effective amount of a therapeutic intervention based at least in part on the prediction regarding the medical outcome of the tissue.
According to another exemplary embodiment of the present disclosure, a method can be provided comprising analyzing on a computer system a feature of a medical image of a tissue and a visual feature of a histopathology sample of the tissue using a specific procedure, wherein the specific procedure is a machine-learning procedure trained on a set of images of cancerous anatomies.
The method of para. [00232] can be provided, wherein the analyzing is of the feature of the medical image of the tissue, the visual feature of the histology sample, and information from an electronic health record corresponding to the tissue.
The method of para. [00232] can be provided, wherein the information from the electronic health record is at least one of (i) a tumor stage of the tissue, or (ii) a status of a molecular biomarker in a subject that provided the tissue.
The method of para. [00232] can be provided, wherein the images of cancerous anatomies are at least one of (i) magnetic resonance images, or (ii) magnetic resonance images of cancerous breasts.
The method of para. [00232] can be provided, wherein the cancerous anatomies are cancerous breasts.
The method of para. [00232] can be provided, further comprising, prior to the analyzing, performing an imaging procedure on the tissue to obtain the medical image of the tissue.
The method of para. [00232] can be provided, wherein the medical image is a magnetic resonance image of at least one of (i) a human breast, (ii) a human body part that is (a) in need of inspection for possible disease, possible cancer, or possible breast cancer, or (b) afflicted with possible disease, possible cancer, or possible breast cancer.
The method of para. [00232] can be provided, further comprising, prior to the analyzing, performing a histopathology procedure on the tissue to obtain the histopathology sample.
The method of para. [00232] can be provided, wherein the histopathology sample is a digitized histopathology slide of the tissue or a human body part that is (i) in need of inspection for possible disease, possible cancer or possible breast cancer, or (ii) afflicted with possible disease, possible cancer or possible breast cancer
The method of para. [00232] can be provided, wherein the visual feature of the histopathology sample is a visual detail that suggests presence of a cancer.
The method of para. [00232] can be provided, wherein the tissue is a human body part that is (i) in need of inspection for possible disease, possible cancer or possible breast cancer, or (ii) afflicted with breast cancer.
The method of para. [00232] can be provided, wherein the analyzing provides a prediction regarding a medical outcome of the tissue.
The method of para. [00243] can be provided, wherein the prediction regarding the medical outcome of the tissue is a likelihood of (i) a relapse of a disease, cancer or breast cancer, or (ii) a favorable response to a therapy for a condition, cancer or breast cancer present in the tissue.
The method of para. [00243] can be provided, further comprising administering to a subject that provided the sample a therapeutic intervention that corresponds to the prediction regarding the medical outcome of the tissue.
According to another exemplary embodiment of the present disclosure, a method can be provided that comprises:
The method of para. [00246] can be provided, wherein the searching procedure is performed for more than one repository of the electronic health records.
The method of para. [00246] can be provided, wherein the breast diagnostic imaging procedures comprise at least one of a radiology or a digital pathology.
The method of para. [00246] can be provided, wherein the input modality is imaging data.
The method of para. [00246] can be provided, wherein at least one portion of the extracted health records is associated with more than one input modality.
The method of para. [00246] can be provided, wherein at least one portion of the extracted health records is associated with at least two of MRI, digital pathology, clinical variables, auxiliary clinical variables, or patient outcomes.
The method of para. [00246] can be provided, wherein the prediction regarding the medical outcome of the tissue is a likelihood of (i) a relapse of a disease or breast cancer, (ii) a favorable response to a therapy for a condition, cancer, or breast cancer present in the tissue.
The method of para. [00246] can be provided, wherein the training of the machine learning procedure comprises pretraining at least one of:
The method of para. [00253] can be provided, wherein the training of the machine learning procedure comprises training the machine learning procedure to map a multi-modal representation to the prediction regarding the medical outcome of the subject based on a multi-modal representation of a plurality of input modalities.
The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures which, although not explicitly shown or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various different exemplary embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary skill in the art. In addition, certain terms used in the present disclosure, including the specification, drawings and claims thereof, can be used synonymously in certain instances, including, but not limited to, for example, data and information. It should be understood that, while these words, and/or other words that can be synonymous to one another, can be used synonymously herein, that there can be instances when such words can be intended to not be used synonymously. Further, to the extent that the prior art knowledge has not been explicitly incorporated by reference herein above, it is explicitly incorporated herein in its entirety. All publications referenced are incorporated herein by reference in their entireties.
This application relates to and claims the benefit of priority from U.S. Provisional Patent Application No. 63/521,262, filed on Jun. 15, 2023, the entire disclosure of which is incorporated herein by reference in its entirety
| Number | Date | Country | |
|---|---|---|---|
| 63521262 | Jun 2023 | US |
