Patent Application
20220117491
This application relates generally to sample analysis and more particularly to multispectral sample analysis using fluorescence signatures.
A material is a substance or a mixture of substances from which an object can be made. Materials, which can be natural or manufactured ones, are widely used by people everywhere. In fact, materials are essential to daily living and even to survival. People wear clothing made from various materials to cover or protect themselves and to keep comfortable and safe. Clothing is also worn to convey information about origin, culture, beliefs, and class. Structures in which people live can be temporary or permanent, depending on purpose, design, and materials used. People travel on or in vehicles manufactured from materials. These vehicles can be powered by people, animals, internal combustion, electricity, or wind, depending on the purpose, destination, and number of people traveling.
Materials that are frequently used to make objects include fabrics, glass, metals plastics, and wood. The materials can be used individually or can be combined with other materials to form compounds, composites, alloys, or blends. The constituents of a material or combination of materials can be identified by studying physical, optical, and other properties. The properties can include material hardness, visual appearance, and weight; physical properties such as state, where the material state includes solid, liquid, gas, or plasma; and other physical properties such as density and magnetic characteristics of the material. The material properties can include chemical properties such as chemical resistance and combustibility. The material properties can include mechanical properties such as malleability, ductility, and strength; and electrical properties such as conductivity and resistivity. The properties of a material can include optical properties such as transmissivity and absorptivity. The physical, chemical, mechanical, electrical, optical, and other responses of a material can be analyzed to characterize and identify unknown materials, since each material has its own unique set of properties.
Analysis and characterization of materials are applicable to many industries including manufacturing, aerospace, and taxonomy, to name but a few. The analysis and characterization of materials is also widely utilized in research applications to identify one or more materials within a sample, to characterize new alloys or compounds, and so on. The analysis and characterization of materials can detect the presence of unexpected materials within a sample. Some applications include identifying contaminants or impurities within materials, where the contaminants cause systems made from the materials to fail. Sophisticated testing procedures and advanced testing techniques can provide detailed information about a material, which can include identification of the chemical composition of the material. This latter class of analysis, based on cutting edge procedures and techniques, can require complex laboratory equipment and advanced training. For example, surface topology and composition of a material can be determined using a scanning electron microscope (SEM), which uses a beam of electrons, while a transmission electron microscope (TEM) can be used in crystalline defect analysis to predict behavior and to find failure mechanisms for materials. Also, X-ray Diffraction (XRD) is used to identify and characterize crystalline materials. These complicated and expensive tests, techniques, and types of equipment, which are usually available only in laboratories, can be used alone or in combination to characterize and identify unknown materials.
Disclosed techniques can be used to characterize and identify materials using multispectral fluorescence signatures. The techniques combine fluorescence spectroscopy and imaging technologies to match measured outputs of Red-Blue-Green (RBG) sensors with material signatures. This technique provides light from a range of wavelengths across the electromagnetic spectrum. A light source excites electrons in molecules of a compound and causes the molecules to emit light or to fluoresce. Multispectral images are captured with a broad-spectrum image sensor. The sensor can include a low-cost RBG sensor. The RBG sensor can employ an integrated, very low-cost Bayer filter. The Bayer filter enables the broad-spectrum image sensor to provide sensitivities to particular wavelengths, including light from frequencies which are visible to the human eye, and light frequencies that are not. Different materials can be distinguished from one another since the different materials reflect and absorb light at different wavelengths. Multispectral imaging can be used to differentiate materials based on their spectral fluorescence signatures, in addition to using their reflection and absorption characteristics. As disclosed, multispectral fluorescence imaging can reduce the complexity, cost, and deployment challenges of using specialized multispectral cameras, elaborate optical filters, and expensive filter wheels, which have orientation and alignment sensitivities. Further, the multispectral fluorescence imaging can be performed without the need for fixed, lab-only equipment placement.
Disclosed techniques address a method for multispectral sample analysis using fluorescence signatures. The analysis can be based on using inexpensive, widely available RBG sensors. At least one fluorescence excitation light wavelength is provided to a material sample. The fluorescence excitation light wavelength signal has a wavelength less than a wavelength of the RGB light wavelength spectrum. The wavelength less than a wavelength of the RGB light wavelength spectrum is substantially between 200 nm and 450 nm. The material sample exhibits fluorescence characteristics along the RGB light wavelength spectrum. Output values of an RGB sensor are measured, where the measuring detects the fluorescence characteristics of the material sample. The fluorescence characteristics are shown in response to the at least one fluorescence excitation light wavelength. An optical bandpass filter to at least one fluorescence excitation light wavelength is added to attenuate wavelengths of the fluorescence excitation light wavelength closest to the RGB light wavelength spectrum. The bandpass filter is centered at 400 nm and has a width of substantially 50 nm. An optical long-pass filter to at least one fluorescence excitation light wavelength is added, where the long-pass filter has a cutoff wavelength less than a wavelength of the RGB light wavelength spectrum. The cutoff wavelength is substantially 30 nm greater than the at least one fluorescence excitation light wavelength. The output of the RGB sensor is compensated based on an analysis of a wavelength response of the RGB sensor. The compensating identifies peak sensitivities for red, green, and blue sensing for the RGB sensor. Output values of an additional RGB sensor are measured, where the measuring detects the fluorescence characteristics of the material sample. The fluorescence characteristics are in response to the at least one fluorescence excitation light wavelength. The RGB sensor and the additional RGB sensor provide a left and a right stereoscopic sensor image. The RGB sensor and the additional RGB sensor are each polarized using polarization filters. Feature matching of the material sample is performed. The indication that is generated enables skin assessment. The skin assessment includes wound assessment, where the wound assessment can include infection detection. The wound assessment can be taken over time to enable a wound care treatment plan. An indication of composition of the material sample is generated. The indication is based on interpreting the output values that were measured. Thermal imaging of the material sample is used to augment the generating. Depth imaging of the material sample also augments the generating.
Various features, aspects, and advantages of various embodiments will become more apparent from the following further description.
The following detailed description of certain embodiments may be understood by reference to the following figures wherein:
Techniques for sample identification based on multispectral sample analysis using fluorescence signatures are disclosed. At least one fluorescence excitation light wavelength is provided to a material sample. The material sample exhibits fluorescence characteristics along the Red-Green-Blue (RGB) light wavelength spectrum. Output values of an RGB sensor are measured. The measuring detects the fluorescence characteristics of the material sample in response to the at least one fluorescence excitation light wavelength. An indication of composition of the material sample is generated. The indication is based on interpreting the output values that were measured.
Fluorophores with different emission spectra can be distinguished based on comparison of their Red-Green-Blue (RGB) emission signals. A fluorescence signal can be spectrally resolved using filters common to many color digital imagers, such as the Red, Green, and Blue Bayer filters integrated in typical, inexpensive RGB sensors that are the basis of common color digital imagers. These sensors generally demonstrate peak blue sensitivity at 400-475 nm, peak green sensitivity at 475-580 nm, and peak red sensitivity at 580-750 nm. A fluorescence excitation source at a wavelength is provided near the edge of, or slightly inside or outside of, the RGB visible light wavelength spectrum, such as at 405 nm. However, it should be noted that the definition of the exact wavelengths of visible light is somewhat subjective. For purposes of discussion, a visible light wavelength range of about 425 nm-725 nm is understood herein, although discrete wavelengths or wavelength ranges are used when possible. The excitation source wavelength can, when used to illuminate a material sample, elicit a fluorescence response from the material sample that can be detected by an RGB sensor. In order to prevent crosstalk from the excitation source into the spectral channels detected by the RGB imager, the excitation source may be outfitted with a bandpass filter. This can be especially useful if the excitation source exhibits a long “red-side” tail into the longer wavelengths detectable by the RGB sensor. Additionally, a long-pass filter placed in front of the RGB sensor can prevent spurious signals from the excitation LED from reaching the RGB sensor.
The low-cost, portable method of multispectral sample analysis disclosed herein uses an ordinary, readily available Red-Green-Blue (RGB) sensor. The RGB sensor typically is mass produced and has applications in low-cost technology that endeavors to detect light waves in the visible spectrum in a standard three-color, RGB palette suitable for digital processing. The RGB sensor typically employs an integrated Bayer filter applied during the manufacturing process of a CMOS, CCD, or similar sensor semiconductor fabrication. The Bayer filter is completely integrated in the sensor and cannot be removed, replaced, or adjusted. When light impinges the surface of an RGB sensor, the underlying photosensors register a signal related to the intensity of the impinging wavelengths as a function of the color of the integrated sensor directly over each photosensor device. The disclosed technology does not require expensive filter wheels, complex optical alignments, or stationary, non-handheld components.
The flow 100 includes providing a fluorescence excitation light to a material sample 110. The excitation light can emanate from a single source or from multiple sources, such as from an incandescent light source, an LED light source, a laser light source, an ultraviolet (UV) light source, an infrared (IR) light source, and so on. The excitation light wavelength can have a wavelength in the ultraviolet light region, which is less than a wavelength of an RGB light wavelength spectrum. The excitation light wavelength can be substantially between 200 nm and 450 nm. The excitation light of one or more wavelengths can illuminate a material sample, which can fluoresce in response. A common, inexpensive CMOS RGB sensor can measure the fluorescence amplitude according to the sensor's designed and manufactured wavelength response. The flow 100 includes measuring output values of an RGB sensor 120, in response to the excitation. The output values of the measured RGB light wavelengths are electrical signals, and thus the RGB sensor translates wavelength intensity to an electrical representation by providing three output values: a red output value, a green output value, and a blue output value. These sensors generally demonstrate peak blue sensitivity at 400-475 nm, peak green sensitivity at 475-580 nm, and peak red sensitivity at 580-750 nm. In order to avoid signal contribution from an excitation LED (or other excitation source) that has a long red tail that may be detected by the spectral channels built into the RGB imager, the excitation source may be outfitted with a bandpass filter that prevents crosstalk. Additionally, a long-pass filter placed in front of the RGB sensor further prevents a spurious signal from the excitation LED. In embodiments, at least one fluorescence excitation light wavelength signal comprises a wavelength less than a wavelength of the RGB light wavelength spectrum. And in embodiments, the wavelength, which is less than a wavelength of the RGB light wavelength spectrum, is substantially between 200 nm and 450 nm.
The flow 100 can include adding an optical bandpass filter 122 between the excitation light and the material sample. A bandpass filter can prevent wavelengths of the excitation light from bleeding into the RGB sensor spectrum and contaminating the results. It should be understood that typical excitation sources will have a spectral energy curve centered at a given wavelength, but that there are usually energy tails at wavelengths other than those of the given wavelength. For example, an excitation source providing a nominal excitation wavelength at 405 nm may have an energy tail in the 1%-10% range at 450 nm, which would contaminate a measurement of an RGB sensor that is sensitive at 450 nm. Other sources, such as a laser excitation source, which are generally more expensive, may be able to provide a narrower wavelength spectrum. The bandpass filter can have a filter width dependent on its characteristics, cost, manufacturing tolerance, and so on. Some embodiments add an optical bandpass filter to at least one fluorescence excitation light wavelength to attenuate wavelengths of the fluorescence excitation light wavelength closest to the RGB light wavelength spectrum. In embodiments, the bandpass filter is centered at 400 nm. And in embodiments, the bandpass filter has a width of substantially 50 nm.
The flow 100 can include adding an optical long-pass filter 124 to the RGB sensor, that is, in between the material sample and the RGB sensor. A long-pass filter can block wavelengths below a cut-off wavelength and allow wavelengths above the cut-off wavelength. For example, a long-pass filter with a cut-off wavelength of 450 nm would prevent excitation wavelengths (e.g., at 405 nm) from contaminating a measurement of an RGB sensor that is sensitive below 450 nm. Of course, no filter is perfect, and the cut-off wavelength may not be a single, well-defined wavelength. Therefore, a bandpass filter and/or a long-pass filter may be used alone or together in order to provide a balance of cost, availability, size, portability, repeatability, etc. Some embodiments add an optical long-pass filter in front of the RGB sensor, which can prevent spurious signals from the excitation LED from reaching the RGB sensor. The long-pass filter is designed to cut off any wavelength which is near or less than a wavelength of the RGB light wavelength spectrum. In embodiments, the cut-off wavelength is substantially 30 nm greater than the at least one fluorescence excitation light wavelength. In embodiments, the long-pass filter has a cutoff wavelength less than a wavelength of the RGB light wavelength spectrum.
The flow 100 can include compensating the output of the RGB sensor 130. The compensating can involve providing a boost or attenuation to electrical output signals of the RGB sensor in order to counteract sensor differences, ambient lighting differences, excitation wavelength spectra differences, and so on. Because various RGB sensors can have various wavelength sensitivities and responses that may vary from sensor to sensor, or from manufacturing lot to manufacturing lot; or may fluctuate due to semiconductor aging, environmental conditions, and so on; compensating can be a key component in achieving sample identification precision. The compensating can be adjusted based on various calibration techniques that are performed before or after an actual sample measurement. Some embodiments compensate the output of the RGB sensor, based on an analysis of a wavelength response of the RGB sensor. In embodiments, the compensating identifies peak sensitivities for red, green, and blue sensing for the RGB sensor.
The flow 100 includes generating an indication composition of the material sample 140. The indication can be generated based on various techniques such as table lookup, graph comparison, machine learning, human interpretation, signature comparison, and the like. The indication can come from a library of RGB output sensor metrics, either compensated or uncompensated. The indication can be useful in many various endeavors as will be discussed shortly. The indication can be based on interpreting the output values 142 of the RGB sensor that were measured and compensated. The indication can be augmented with thermal imaging 144 or depth imaging 146. A multispectral sample analysis can combine the indication with thermal imaging and depth imaging via stereoscopy, LIDAR, Time of Flight, and so on, to determine how sample position in a scene determines signal intensity, which can subsequently be used to improve indication specificity. For example, if x is a distance between the sample and the RGB sensor, which could have integrated or discrete focusing lenses included, then to correct for photon density at x, the raw sample image at x can be compared to a diffuse reflectance standard at x. Some embodiments augment the generating with thermal imaging of the material sample. And some embodiments augment the generating with depth imaging of the material sample.
The flow 100 includes adding a stereoscopic sensor 148. An additional RGB sensor can be added to provide another angle from which an RGB sensor provides output. The additional RGB sensor enables stereoscopic imaging of the material sample. Output values of the additional RGB sensor can be used along with output values of the RGB sensor to indicate composition, determine features, compare features over time, and so on. Some embodiments include measuring output values of an additional RGB sensor, wherein the measuring detects the fluorescence characteristics of the material sample, and wherein the fluorescence characteristics are in response to the at least one fluorescence excitation light wavelength. In embodiments, the RGB sensor and the additional RGB sensor provide a left and a right stereoscopic sensor image.
The flow 100 includes using the RGB sensor and the additional RGB sensor with polarization 150. Polarization filters can be placed over, on, or in front of an RGB sensor to attenuate photon detection of one polarization, but to allow mostly unattenuated photon detection for another polarization. The polarization filters can be placed over the RGB sensors 90° out of phase with each other. Thus, one RGB sensor detects primarily “parallel” light wavelengths, and the other RGB sensor detects primarily “perpendicular” light wavelengths. The polarization filters can be chosen such that they lose their polarization effectiveness above a certain wavelength, for example, above a 700 nm wavelength. Thus, detecting polarized photons below a 700 nm wavelength and non-polarized photons above a 700 nm wavelength is enabled. In embodiments, the RGB sensor and the additional RGB sensor are each polarized using polarization filters. Some embodiments include feature matching of the material sample. For fluorescence photons, which are not inherently polarized as emitted from the material sample, and for wavelengths over the effective polarization wavelength of 700 nm (e.g., at 940 nm), features detectable in both sensors are the same. Thus, at 940 nm, the polarizers that sit in front of all of the LED's do not polarize the light and as a result, the specular reflections bleed through. And in the case of fluorescence photons, which are not polarized, there is no specular reflection. Thus, feature mapping for fluorescence photons is enabled through the polarization filters.
Various steps in the flow 100 may be changed in order, repeated, omitted, or the like without departing from the disclosed concepts. Various embodiments of the flow 100 can be included in a computer program product embodied in a non-transitory computer readable medium that includes code executable by one or more processors.
The flow 200 includes providing an absorption excitation light wavelength 210 and a second absorption excitation light wavelength 212. The two excitation light wavelengths can provide information on absorption characteristics. The light wavelengths can be within the visible spectrum and/or outside of the visible spectrum. The flow 200 can include providing a further absorption excitation light wavelength 216. Each of the excitation wavelengths can provide a data point or points related to the absorption characteristics of the material sample (e.g., at least three data points, one for each excitation wavelength). The absorption characteristics can be ascertained by measuring the output values of an RGB sensor 220. Each output red, green, and blue of the RGB sensor provides a relative measurement of the material sample absorption while the material sample is irradiated with the excitation light wavelengths.
The flow 200 can include spacing the wavelengths of the excitations at least 100 nm apart 222. An excitation wavelength can be substantially at a certain wavelength when its spectral energy peak encompasses that wavelength within about 10% of the wavelength. This is illustrated by excitations 622, 624, and 626 of
The flow 200 includes enabling water identification 232. Unlike most biochromes, water exhibits a monotonically increasing absorption characteristic across an excitation profile as the excitation wavelength increases. Because water is such an integral component of living tissue, water identification can be very useful. Isolating water absorption can be performed by monitoring absorption at 800-1000 nm and comparing to absorption at longer wavelengths. Absorption by most chromophores found in nature decreases with increasing wavelength; however, in the case of water the opposite is true. A light source with peak intensity from 800-1000 nm can be used to generate an absorption signal based on a comparison to a diffuse reflectance standard:
where the raw image is the output of the RGB sensor with excitation light illumination as described herein, the dark image is the output of the RGB sensor with no excitation light illumination and only ambient lighting conditions, and the DR (diffuse reflectance) standard is a known and characterized sample that provides a baseline output of the RGB sensor with excitation light illumination.
Thus, a method for multispectral sample analysis is disclosed comprising: providing at least two excitation light wavelengths to a material sample, wherein the material sample exhibits absorption characteristics along the Red-Green-Blue (RGB) light wavelength spectrum; measuring output values of an RGB sensor, wherein the measuring detects the absorption characteristics of the material sample, and wherein the absorption characteristics are determined in response to the at least two excitation light wavelengths; and generating an indication of composition of the material sample, wherein the indication is based on interpreting the output values that were measured. Some embodiments include providing a third excitation light wavelength and measuring an additional output value of the RGB sensor. In embodiments, the at least two excitation light wavelengths and the third excitation light wavelength are each at a wavelength substantially 200 nm apart. In embodiments, the providing at least two excitation light wavelengths and the providing a third excitation light wavelength enables water identification. In embodiments, the water identification comprises identifying a predominately monotonically increasing absorption at longer wavelengths. Predominately monotonically can indicate at least an order of magnitude difference at two points. In embodiments, the providing at least two excitation light wavelengths and the providing a third excitation light wavelength enables biochrome identification. And some embodiments include providing at least one further additional excitation light wavelength and measuring a further additional output value of the RGB sensor.
The flow 200 includes using an additional, stereoscopic RGB sensor 234. The RGB sensor and the additional RGB sensor can provide a stereoscopic image of the material sample. Both the RGB sensor and the additional RGB sensor can be used with polarization 236. The RGB sensors can have polarization filters inserted over or in front of them to provide a measure of polarization in the images. For example, one sensor can be “polarized” in a vertical direction, while the other sensor can be “polarized” in a horizontal direction, thus providing 90° “cross-polarization” for the stereoscopic imaging. This cross-polarization allows for the isolation of specularly reflected, polarized photons based on comparison of the images taken from the two sensors. Photons that undergo multiple scattering events deeper in the skin lose their polarization and contribute equally to the parallel and perpendicularly polarized signal. Deeper scattering tends to take place in, for example, the dermis, where randomly oriented collagen fibers primarily contribute to the loss of polarization.
The polarization filters can be chosen such that they lose their polarization effectiveness above a certain wavelength, for example, above a 700 nm wavelength. Thus, detecting polarized photons below a 700 nm wavelength and non-polarized photons above a 700 nm wavelength is enabled. In embodiments, the RGB sensor and the additional RGB sensor are each polarized using polarization filters. Some embodiments include feature matching of the material sample. For example, absorption images taken at 460 nm, 523 nm, and 660 nm when the left camera is polarized parallel to the LEDs and the right camera is polarized perpendicular to the LEDs pose a problem: specular reflection features aren't visible in both images. For stereomatching feature identification, the same features are required in both images. However, for wavelengths over the effective polarization wavelength of 700 nm (e.g., at 940 nm), features detectable in both sensors are the same. Thus, at 940 nm, the polarizers that sit in front of all of the LED's do not polarize the light and the specular reflections bleed through. Thus, feature mapping for the absorption case, that is, the photons that are not absorbed and are reflected back to the sensors, is enabled through the polarization filters.
Various steps in the flow 200 may be changed in order, repeated, omitted, or the like without departing from the disclosed concepts. Various embodiments of the flow 200 can be included in a computer program product embodied in a non-transitory computer readable medium that includes code executable by one or more processors.
The system block diagram 300 can include one or more optical filters 320 on the source side of a material sample 330. That is, the one or more excitation wavelengths 310, 312, and 314 can be conditioned by the one or more optical filters 320 such that the illuminating light from the excitation wavelengths is affected by the filters before it reaches the material sample 330. These filters do not affect the fluorescence emissions of the material sample that are detected by an RGB sensor, based on the stimulation of the one or more excitation wavelengths. The filter 320 can be a bandpass filter. The one or more excitation wavelengths, as conditioned by any intervening filters 320, then impinge on a material sample 330, resulting in a fluorescence emission from the sample that is detected by RGB measurement block 340. Note that before RGB measurement block 340, the system block diagram 300 indicates light transmission, as denoted by the dashed lines among blocks 310, 312, 314, 320, 330, and 340. The output of RGB measurement block 340, as well as the signals between subsequent blocks 350 and 360, are electrical signals, as denoted by the solid lines. Optionally, an additional optical filter (not shown) can be placed between the material sample 330 and the RGB measurement 340. The additional optical filter can be a long-pass filter.
The electrical output of RGB measurement block 340 can be compensated by compensation block 350. Compensation can involve providing a boost or attenuation to electrical signals indicating a certain magnitude of a particular light wavelength in order to counteract sensor differences, ambient lighting differences, excitation wavelength spectra differences, and so on. The compensation block 350 can be adjusted based on various calibration techniques that are performed before or after an actual sample measurement. The output of compensation block 350 can enable generation of an indication 360 of a composition of a material sample. Analysis of the output of compensation block 350 (or directly from RGB measurement block 340) can enable generation of an indication of composition, based on the output of block 350 (or directly from block 340) using various methods such as table lookup, graph comparison, machine learning, human interpretation, signature comparison, and the like.
The indication can enable skin assessment 420. The skin assessment can involve predicting the onset of skin conditions such as psoriasis, which can be distinguished based on fluorescence from fluorophores such as melanin, elastin, collagen, keratin, and flavoprotein. Other skin conditions, such as eczema and acne, can also be predicted. In addition, skin hydration can be assessed using the disclosed techniques. The skin assessment can include feature identification. The indication can enable wound assessment 422. The wound assessment can be based on collecting a variety of images at different excitation wavelengths and spatially registering the images using micro- or macro-scale features, skin and wound edges, fiducial marks, reference standards for alignment, corresponding biological features, and the like. Feature recognition can be accomplished using Laplace of Gaussians, difference of Gaussians, Hessian-Laplace, scale invariant feature transform (SIFT), multi-scale-oriented patches (MOPS), or other image processing techniques for local feature description. Once corresponding features on images are identified, the registration technique can use translation, rigid body, rotation, or affine transformation methods to register multiple images collected at different wavelengths. A pixel-by-pixel registration allows for the images to be digitally processed in order to identify biological features, to perform calculations which isolate or enhance the biological signals, and/or to assess wound healing. Further analysis can enable algorithmic identification of infection. In embodiments, the wound assessment includes infection detection. In embodiments, the skin assessment includes wound assessment. In embodiments, the wound assessment is taken over time. In embodiments, the wound assessment taken over time enables a wound care treatment plan. In embodiments, the skin assessment is updated using temporal change feature matching, that is, by comparing identified features in the wound to determine how they are changing temporally (i.e., with the passage of time). The temporal change can occur over two or more healthcare clinical sessions. At least one of the two or more healthcare clinical sessions can be self-administered.
As discussed previously, the indication can enable biochrome identification 430 and water identification 432. In addition, the indication can enable infection detection 434 or respiratory infection detection 436. Host metabolism plays a vital role in viral infections. Energy yielding metabolic pathways are repurposed by the virus to support viral replication. High concentrations of nicotinamide adenine dinucleotide+hydrogen (NADH) and flavins are indicative of such infections. The indication can be generated by isolating signals from NADH and flavins by collecting fluorescence photons in the R, G, and B channels, respectively, and exciting at or near 400 nm. This approach further isolates features in an image that can be attributed to the presence of flavins and NADH by taking the normalized ratio, where normalization is based on excitation flux, integration time, and channel sensitivity of the green channel signal to the blue channel signal, and isolating based on pixels that yield a ratio value indicative of the presence of NADH and/or flavins.
In addition, abnormal concentrations of porphyrin, which can be detected using the disclosed concepts, have been observed in serum from COVID-19 patients. Other respiratory related infections, such as sinusitis, are more prevalent with a common cold than with influenza. These infections can be analyzed based on the fact that signatures of sinusitis, such as fluid in the sinuses, can increase the indication precision to distinguish between respiratory infection types. Furthermore, common cold viruses usually do not cause substantial damage to the airway epithelium, whereas influenza and COVID-19 can damage cells in the respiratory epithelium. In fact, a broad variety of respiratory pathogens, including rhinoviruses, coronaviruses, and the like can adversely affect cells. Redness and inflammation associated with such cellular damage can be detected using the disclosed techniques. By applying the disclosed techniques when looking into a patient's throat and taking images to measure fluorescence, absorption and thermal radiation from the throats of patients with possible infection from respiratory viruses such as SARS-CoV-2, Influenza A and Influenza B can be detected. Such methods can also facilitate telemedicine diagnostics. In embodiments, the indication enables infection detection. In embodiments, the infection detection is based on biochrome identification. In embodiments, the indication enables respiratory infection detection. In embodiments, the respiratory infection detection comprises influenza detection. In embodiments, the influenza detection comprises COVID-19 detection.
This technology isolates signals from infection-associated biochromes, such as Porphyrin and Pyoverdine, by holding an excitation wavelength constant and collecting signals from progressively longer wavelength emission channels. This action is performed at each pixel in an image. In one embodiment, fluorescence is collected by exciting wavelengths in the blue/UV region of the spectrum such that the peak of the spectral distribution of the excitation source is at a lower wavelength (higher energy) than what is typically detected by the sensor (CMOS or CCD as examples) that is being used for detecting photons and generating an image.
The indication can enable residual cancer detection 438. Autofluorescence imaging is enabled by the disclosed concepts and has been used to diagnose oral cancer, breast cancer, lung cancer, skin cancer, brain cancer, and others. Autofluorescence from NADH has been cited as one possible biomarker for targeting cancer. Similarly, fluorescence from dense connective tissue (extracellular matrix, etc.) associated with tumor can be used to delineate tumor boundaries. In addition, such techniques can enable detection of residual cancer during surgery. In embodiments, the indication enables residual cancer detection. In embodiments, the residual cancer detection occurs during oncological surgery.
The indication can enable food recognition, food quality, or food safety 440. Common food borne pathogens include E. coli, Salmonella, Listeria, Cyclospora, and Hepatitis A. Disclosed techniques can enable fast detection of food borne pathogens in order to avoid distribution of contaminated foods. Authentication, quality, and possible adulteration of food must be monitored for distribution and consumption. For example, liquor, wine, and beer inspection can be performed by analyzing both water content and the presence of fluorescent compounds. Fluorescent compounds such as polyphenols, flavonoids, stilbenes, tannins, coumarins, and fluorescent amino acids are key markers of authenticity and quality. In some embodiments, two or three excitation LEDs at different blue and UV wavelengths may be employed for determining a shift in emission resulting from a change in excitation frequency. Such techniques can be used in plant food quality analysis, milk quality analysis, fruit quality analysis, coffee quality analysis, as well a protein quality analysis of products as varied as beef and sashimi, to name just a few. Other applications include monitoring the progress of fermentation, such as malolactic fermentation, for the deacidification of red wines. In-line monitoring of the fermentation process can also be applied to fermentation processes in which yeast or bacteria are programmed to produce a specific chemical such as THC and CBD. In addition, monitoring caloric intake can be enabled by food composition and rough, overall portion size identification. In embodiments, the indication enables food recognition, food quality, or food safety identification. In embodiments, the food quality detects food adulteration. And in embodiments, the food quality monitors progression of fermentation. In embodiments, the indication of composition enables food identification.
The indication can enable agricultural yield optimization 442. Especially in automated indoor farming, which is poised to assume a significant burden of the food supply, the disclosed techniques can enable identification of crop ripeness, crop water sufficiency, crop fertilization sufficiency, crop disease detection, and so on. This approach can enable minimized use of insecticides and herbicides while optimizing crop yield. In addition, a robot- or drone-based approach to agricultural optimization is feasible due to the portable attributes of the disclosed techniques. In embodiments, the indication enables agricultural yield optimization. In embodiments, providing excitation and measuring RGB sensor output values are accomplished using drone technology. As discussed throughout, when excitation wavelengths of light illuminate a target material, certain molecules respond with a fluorescence signature. The magnitude of such a signature can provide an indication of the amount, distribution, concentration, purity, etc. of the fluorescence molecule. For example, using the described techniques on, say, a carrot, can provide insight into the amount of beta carotene present in the carrot sample. Similarly, molecules such as THC or CBD can be monitored in situ, that is, when a crop with such molecules present is still planted in a field and yet to be harvested. Thus the indication of composition can enable in situ crop monitoring. The crop monitoring can include evaluation of crop disease, crop ripeness, or crop quality. The evaluation of crop quality includes determining fluorescent molecule concentration.
The indication can have applications in law enforcement and can enable a field sobriety evaluation 444 for an individual. A contactless evaluation using the disclosed techniques can determine the need for a more invasive breathalyzer test. In addition to visual indicators such as enlarged pupils and eye movement that is faster than normal, measured amounts of vasoconstriction and vasodilation, depending on a level of intoxication, can be enabled using the indication. In embodiments, the indication enables field sobriety evaluation of individuals. In embodiments, the field sobriety evaluation of individuals is accomplished in a contactless manner. The indication can have further applications in dental care. The indication can enable an oral hygiene evaluation 446 for an individual. This can include detecting plaques, gingivitis, and other dental abnormalities using multispectral imaging and fluorescence. Thus in embodiments, the indication enables oral hygiene evaluation.
In the graph 500, an x-axis indicating wavelength 510 is provided. Increasing wavelength from left to right indicates decreasing frequency of light waves and a traversal from the ultraviolet spectrum, roughly sub-400 nm, through the blue, green, and red wavelength regions, roughly 450 nm, 550 nm, and 650 nm, respectively, to the infrared wavelength band, which is roughly greater than 750 nm. It should be noted that an exact wavelength definition of a particular color is somewhat arbitrary and dependent on the sensor type. For example, the cones of a human eye roughly sense RGB signals using three cone types, but they are generally distributed differently from a typical CMOS RGB sensor's output. However, maintaining a consistent definition for a given system is generally required in order to provide consistent sample indications. The graph 500 also includes a left y-axis of absorption amount 512 from one to ten and a right y-axis of transmission amount 514 from zero to one.
The graph 500 includes excitation wavelength 522. The excitation wavelength 522 is centered substantially at 405 nm in the ultraviolet light wavelength spectrum. Note that wavelength 522 is a relatively narrow excitation, but that due to practical considerations, energy tails of the excitation wavelength can sometimes extend up toward the visible light RGB spectrum at 450 nm and above. To prevent bleed-over into the RGB spectrum, a bandpass filter, indicated by transmission spectrum 524, can be included. The bandpass filter can help attenuate excitation wavelengths outside of the band, such as a 50 nm bandpass filter centered at 400 nm. To further prevent bleed-over into the RGB spectrum, a long-pass filter, indicated by absorption spectrum 526, can be included. It should be noted that the bandpass optical filters can be placed between an excitation source and a sample, and the long-pass optical filter can be placed between the sample and an RGB sensor. In this manner, the excitation wavelength does not “bleed over” and affect the fluorescence measurements of wavelengths being emitted by the stimulated sample.
The graph 500 includes RGB sensor characteristics, such as sensor characteristic 532, indicative of the “R” or red output of an RGB sensor, sensor characteristic 534, indicative of the “G” or green output of an RGB sensor, and sensor characteristic 536, indicative of the “B” or blue output of an RGB sensor. The RGB outputs represented by characteristics 532, 534, and 536 can be used directly or can be compensated (as discussed elsewhere) to enable generation of an indication of material composition.
The graph 600 illustrates three excitation wavelengths for sample illumination. Excitation wavelength 622 is substantially centered at a wavelength of about 523 nm; excitation wavelength 624 is substantially centered at a wavelength of about 660 nm; and excitation wavelength 626 is substantially centered at a wavelength of about 940 nm. Thus, the three excitation wavelengths, wavelength 622, wavelength 624, and wavelength 626, are spaced at least 100 nm apart over an extended visible light spectrum. The sharp, bell curve shape of the excitations provides for little to no overlap of those excitation wavelengths. Also, it can be noted that 940 nm light is sometimes considered to be near-infrared (NIR) wavelength light. However, most silicon-based CMOS sensors detect 940 nm light. In some usage scenarios, a short-pass filter is applied to prevent noise from NIR photons if that wavelength is not being used by an application. Nonetheless, a 940 nm wavelength can be considered part of an extended visible light spectrum and included when discussing RGB sensor usage.
The graph 600 includes various biochrome and water absorption characteristics, such as a hemoglobin (Hgb) absorption characteristic 632 and a water absorption characteristic 634. By taking the value of each absorption characteristic line on graph 600 at each of the three excitation wavelengths 622, 624, and 626, a tri-valued metric can be determined. Notably, while many of the biochrome absorption characteristics wander about with no simple trend across increasing wavelength, such as is observed for Hgb absorption characteristic 632, the water absorption characteristic 634 displays a monotonically increasing metric across increasing wavelength excitations 622, 624, and 626, which metric increases close to three orders of magnitude across the excitations.
The graph 600 shows other absorption characteristics, such as melanin absorption characteristic 640, fat absorption characteristic 639, oxygenated hemoglobin (oxyHgb) absorption characteristic 638, and collagen absorption characteristic 636. The graph 600 thus illustrates a method for multispectral sample analysis comprising: providing at least two excitation light wavelengths to a material sample, wherein the material sample exhibits absorption characteristics along the Red-Green-Blue (RGB) light wavelength spectrum; measuring output values of an RGB sensor, wherein the measuring detects the absorption characteristics of the material sample, and wherein the absorption characteristics are in response to the at least two excitation light wavelengths; and generating an indication of composition of the material sample, wherein the indication is based on interpreting the output values that were measured.
The system 700 can include a providing component 720. The providing component 720 can be used to provide light excitation wavelengths directed toward a material sample undergoing analysis. The light excitation provided can come from various different sources including an incandescent light source, an LED light source, a laser light source, and so on. The light source or sources can emit a narrow spectrum of light at primarily one wavelength, at primarily two or more wavelengths, across a broad spectrum of multiple wavelengths, in the visible spectrum, in the infrared spectrum, in the ultraviolet spectrum, and so on. The excitation wavelengths can be targeted towards material sample fluorescence or material sample absorption. A fluorescence excitation light wavelength signal can have a wavelength less than a wavelength of the RGB light wavelength spectrum. The wavelength less than a wavelength of the RGB light wavelength spectrum can be substantially between 200 nm and 450 nm. A wavelength substantially between 200 nm and 450 nm can indicate that a high percentage of the excitation light energy, for instance at least 90%, is contained within the 200 nm to 450 nm wavelength region. The term “substantially” reflects an understanding that no real world, physically-based system can be described in exact terms, and therefore it is accurate and efficient to describe light wavelengths in a real system using “substantially” as a modifier.
The system 700 can include a measuring component 750. The measuring component 750 can provide a digital or analog signal output related to the magnitude of incoming light wavelengths from a sample. The measuring component 750 can comprise an RGB sensor. The output from the RGB sensor of the measuring component can be processed using various signal processing techniques. For example, the measuring component output can be compensated to account for naturally occurring manufacturing differences in the RGB sensor by completing a calibration step before the material sample is analyzed.
The system 700 can include a generating component 760. The generating component 760 can provide analysis of the RGB sensor output from the measuring component 750 to provide indication of composition of the material sample, based on interpreting the output values that were measured. The interpreting the output values to provide indication of composition can be performed using various methods such as table lookup, graph comparison, machine learning, human interpretation, signature comparison, and the like. The indication of composition can be useful for enabling medical evaluation such as skin assessment; wound assessment; wound assessment over time; treatment planning for wound care; infection detection; biochrome identification; respiratory infection detection; influenza detection; COVID-19 detection; residual cancer detection; oncological surgery residual cancer detection; oral hygiene detection such as detecting plaques, gingivitis, and other dental abnormalities; and so on. Further, the indication of composition can have applications in food recognition, food quality assessment, or food safety evaluation, detecting food adulteration, monitoring progression of fermentation, optimizing agricultural yield, and enabling field sobriety evaluation of individuals, to name just a few.
The system 700 can include a computer program product embodied in a non-transitory computer readable medium for multispectral sample analysis, the computer program product comprising code which causes one or more processors to perform operations of: providing at least one fluorescence excitation light wavelength to a material sample, wherein the material sample exhibits fluorescence characteristics along the Red-Green-Blue (RGB) light wavelength spectrum; measuring output values of an RGB sensor, wherein the measuring detects the fluorescence characteristics of the material sample, and wherein the fluorescence characteristics are in response to the at least one fluorescence excitation light wavelength; and generating an indication of composition of the material sample, wherein the output is based on interpreting the output values that were measured.
Each of the above methods may be executed on one or more processors on one or more computer systems. Embodiments may include various forms of distributed computing, client/server computing, and cloud-based computing. Further, it will be understood that the depicted steps or boxes contained in this disclosure's flow charts are solely illustrative and explanatory. The steps may be modified, omitted, repeated, or re-ordered without departing from the scope of this disclosure. Further, each step may contain one or more sub-steps. While the foregoing drawings and description set forth functional aspects of the disclosed systems, no particular implementation or arrangement of software and/or hardware should be inferred from these descriptions unless explicitly stated or otherwise clear from the context. All such arrangements of software and/or hardware are intended to fall within the scope of this disclosure.
The block diagrams and flowchart illustrations depict methods, apparatus, systems, and computer program products. The elements and combinations of elements in the block diagrams and flow diagrams, show functions, steps, or groups of steps of the methods, apparatus, systems, computer program products and/or computer-implemented methods. Any and all such functions—generally referred to herein as a “circuit,” “module,” or “system”—may be implemented by computer program instructions, by special-purpose hardware-based computer systems, by combinations of special purpose hardware and computer instructions, by combinations of general-purpose hardware and computer instructions, and so on.
A programmable apparatus which executes any of the above-mentioned computer program products or computer-implemented methods may include one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, programmable devices, programmable gate arrays, programmable array logic, memory devices, application specific integrated circuits, or the like. Each may be suitably employed or configured to process computer program instructions, execute computer logic, store computer data, and so on.
It will be understood that a computer may include a computer program product from a computer-readable storage medium and that this medium may be internal or external, removable and replaceable, or fixed. In addition, a computer may include a Basic Input/Output System (BIOS), firmware, an operating system, a database, or the like that may include, interface with, or support the software and hardware described herein.
Embodiments of the present invention are limited to neither conventional computer applications nor the programmable apparatus that run them. To illustrate: the embodiments of the presently claimed invention could include an optical computer, quantum computer, analog computer, or the like. A computer program may be loaded onto a computer to produce a particular machine that may perform any and all of the depicted functions. This particular machine provides a means for carrying out any and all of the depicted functions.
Any combination of one or more computer readable media may be utilized including but not limited to: a non-transitory computer readable medium for storage; an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor computer readable storage medium or any suitable combination of the foregoing; a portable computer diskette; a hard disk; a random access memory (RAM); a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, or phase change memory); an optical fiber; a portable compact disc; an optical storage device; a magnetic storage device; or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
It will be appreciated that computer program instructions may include computer executable code. A variety of languages for expressing computer program instructions may include without limitation C, C++, Java, JavaScript™, ActionScript™, assembly language, Lisp, Perl, Tcl, Python, Ruby, hardware description languages, database programming languages, functional programming languages, imperative programming languages, and so on. In embodiments, computer program instructions may be stored, compiled, or interpreted to run on a computer, a programmable data processing apparatus, a heterogeneous combination of processors or processor architectures, and so on. Without limitation, embodiments of the present invention may take the form of web-based computer software, which includes client/server software, software-as-a-service, peer-to-peer software, or the like.
In embodiments, a computer may enable execution of computer program instructions including multiple programs or threads. The multiple programs or threads may be processed approximately simultaneously to enhance utilization of the processor and to facilitate substantially simultaneous functions. By way of implementation, any and all methods, program codes, program instructions, and the like described herein may be implemented in one or more threads which may in turn spawn other threads, which may themselves have priorities associated with them. In some embodiments, a computer may process these threads based on priority or other order.
Unless explicitly stated or otherwise clear from the context, the verbs “execute” and “process” may be used interchangeably to indicate execute, process, interpret, compile, assemble, link, load, or a combination of the foregoing. Therefore, embodiments that execute or process computer program instructions, computer-executable code, or the like may act upon the instructions or code in any and all of the ways described. Further, the method steps shown are intended to include any suitable method of causing one or more parties or entities to perform the steps. The parties performing a step, or portion of a step, need not be located within a particular geographic location or country boundary. For instance, if an entity located within the United States causes a method step, or portion thereof, to be performed outside of the United States then the method is considered to be performed in the United States by virtue of the causal entity.
While the invention has been disclosed in connection with preferred embodiments shown and described in detail, various modifications and improvements thereon will become apparent to those skilled in the art. Accordingly, the foregoing examples should not limit the spirit and scope of the present invention; rather it should be understood in the broadest sense allowable by law.
This application claims the benefit of U.S. provisional patent application “Multispectral Sample Analysis Using Fluorescence Signatures” Ser. No. 63/132,541, filed Dec. 31, 2020. This application is also a continuation-in-part of U.S. patent application “Skin Diagnostics Using Optical Signatures” Ser. No. 17/155,141, filed Jan. 22, 2021, which claims the benefit of U.S. provisional patent applications “Systems and Methods for Wound Care Diagnostics and Treatment” Ser. No. 62/964,969, filed Jan. 23, 2020, and “Multispectral Sample Analysis Using Fluorescence Signatures” Ser. No. 63/132,541, filed Dec. 31, 2020. Each of the foregoing applications is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63132541 | Dec 2020 | US | |
| 62964969 | Jan 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17155141 | Jan 2021 | US |
| Child | 17564298 | US |
