Patent Application
20190317079
The present disclosure relates generally to image processing. More particularly, the present disclosure relates to a system and method for detecting volatile organic compounds in an analyte.
Volatile organic compounds (“VOCs”) are organic compounds that can change, often easily, into a gaseous state at ambient temperatures and pressure conditions. VOC molecules generally include hydrogen, oxygen, fluorine, chlorine, bromine, sulfur and nitrogen in addition to carbon, and can be found in nature, such as in oil and gas fields, and as naturally occurring rubber and resin in plants. VOCs can also be released from man-made products such as paints, inks, solvents, glues, perfumes, cleaning products, landfills, treatment plants, fuel combustion and from effluents from factories and refineries. Many VOCs are hazardous air pollutants and can be lethal in high concentrations. VOCs pose serious health concerns as they can exhibit toxic, carcinogenic, mutagenic and neurotoxic properties. Short term VOC exposure may cause fatigue, headache, dizziness, itchy eyes, skin irritation, shortness of breath, and other symptoms, while long term exposure may result in damage to the nervous system and organs such as the liver and kidneys. Public health agencies have created regulations and solvent emissions directives that limit the emissions of VOCs into indoor and outdoor air.
The presence of VOCs in human breath and urine has great potential for medical, toxicological, forensic and homeland security applications. Studies have indicated that VOCs in humans correlate well with some diseases, metabolic disorders and frequent exposure to toxic environmental contaminants including chemical, biological and radiological agents.
VOCs can also be found in warfare agents such as chemical weapons and explosives. These can include nerve agents, vesicants and poisons. Explosives and chemical weapons are highly reactive and can include compounds such as phosgene, cyanides, organic peroxides, nitrated aromatic and aliphatic compounds, fuel oxidizer mixtures, and the like. These compounds and products created from their degradation over time may release VOCs. This property can be exploited to detect chemical weapons and explosives if the constituting VOCs can be detected and identified.
Existing technologies for detecting VOCs, such as electronic noses, have limitations; for example: sensitivity to oxygen containing compounds, ammonia, carbon-dioxide, high operating temperatures, sensitive to humidity, bulky, complex setup, requirement of controlled environment, limited shelf life, lack of reusability due to permanent denaturing of certain elements upon single exposure to certain VOCs, complex circuitry, and baseline drift, among others. Such technologies can have high operational expenses, maintenance costs, and training costs, as well as complex operational procedures making them unsuitable for large scale deployment.
Accordingly, a system and method for detecting VOCs is desired that alleviates limitations of current methods and systems, such as through the application of machine learning techniques to image data.
In an aspect, there is provided a method of identifying an analyte based on the presence of at least one volatile organic compound (“VOC”) in the analyte, comprising: receiving image data from a sensor array after the sensor array has been exposed to the analyte, the sensor array comprising at least one sensor configured to respond to the presence of the at least one VOC in the analyte; processing the image data to derive one or more input image features; and using a trained machine learning classification technique, detecting the at least one VOC and classifying the analyte based on the one or more input image features, the machine learning classification technique trained using one or more reference images of known analytes.
In a particular case, the sensor array comprises a colorimetric sensor array of a plurality of colorimetric sensors, each colorimetric sensor changing in color or in color intensity when exposed to the VOC present in the analyte.
In another case, receiving the image data comprises repeatedly receiving image data of the sensor array at a series of time intervals.
In yet another case, processing the image data to derive the one or more input image features comprises comparing image data in each of the images in the image series to the one of the reference images to generate comparison image data comprising color differences, the input image features comprising the comparison image data.
In yet another case, for each image in the image series, processing the image data comprises: performing edge detection on the image data; performing a Hough circle transform on the image data; detecting an image rotation angle; and where there is a previous image in the image series, aligning the image with the previous image.
In yet another case, processing the image data comprises applying adaptive thresholding to produce a binarized image of the image data.
In yet another case, processing the image data further comprises performing a Hough Circle Transform to detect a blob circle on the binarized image and to segment color blobs on the binarized image.
In yet another case, processing the image data further comprises predicting missing a missing color blob using geometric interpolation.
In yet another case, the one or more input image features comprise color features comprising at least one of global mean, global mode, inner mean, and inner mode.
In yet another case, the one or more input image features comprise textural features comprising at least one of co-occurrence matrix, angular second moment, contrast, correlation, entropy, Hellinger Distance, and Hausdorff Distance.
In yet another case, the machine learning classification technique comprises one of principal component analysis, support vector machines, stacked auto encoders, multi-layer perceptrons, recurrent neural networks, or deep learning neural networks.
In another aspect, there is provided a system for identifying an analyte based on the presence of at least one volatile organic compound (“VOC”) in the analyte, the system in communication with an image acquisition device, the system comprising one or more processors in communication with a memory, the one or more processors configured to execute: an image processing module to: receive image data from a sensor array on the image acquisition device after the sensor array has been exposed to the analyte, the sensor array comprising at least one sensor configured to respond to the presence of the at least one VOC in the analyte; and process the image data to derive one or more input image features; and a classification module to, using a trained machine learning classification technique, classify the analyte based on the one or more input image features, the machine learning classification technique trained using one or more reference images of known analytes.
In a particular case, the sensor array comprises a colorimetric sensor array of a plurality of colorimetric sensors, each colorimetric sensor changing in color or in color intensity when exposed to the VOC present in the analyte
In another case, receiving the image data comprises repeatedly receiving image data of the sensor array at a series of time intervals.
In yet another case, processing the image data to derive the one or more input image features comprises comparing image data in each of the images in the image series to the one of the reference images to generate comparison image data comprising color differences, the input image features comprising the comparison image data.
In yet another case, processing the image data comprises applying adaptive thresholding to produce a binarized image of the image data.
In yet another case, processing the image data further comprises performing a Hough Circle Transform to detect a blob circle on the binarized image and to segment color blobs on the binarized image.
In yet another case, the one or more input image features comprise color features comprising at least one of global mean, global mode, inner mean, and inner mode.
In yet another case, the one or more input image features comprise textural features comprising at least one of co-occurrence matrix, angular second moment, contrast, correlation, entropy, Hellinger Distance, and Hausdorff Distance.
In yet another case, the machine learning classification technique comprises one of principal component analysis, support vector machines, stacked auto encoders, multi-layer perceptrons, recurrent neural networks, or deep learning neural networks.
These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.
The features of the invention will become more apparent in the following detailed description in which reference is made to the appended drawings wherein:
Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.
Various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: “or” as used throughout is inclusive, as though written “and/or”; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender; “exemplary” should be understood as “illustrative” or “exemplifying” and not necessarily as “preferred” over other embodiments. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.
Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.
The following relates generally to image processing, and more particularly to the detection of volatile organic compounds (“VOCs”) in an analyte through application of machine learning techniques to image data.
VOCs can be detected based on different principles and interactions between organic compounds and sensor elements. For example, some devices can detect total ppm concentrations without any specificity whereas others can predict molecular structure of a VOC with reasonable accuracy. Some currently commercially available VOC sensors include electrochemical sensors, metal oxide semiconductor sensors, infrared sensors, thermal sensors, flame ionization sensors, and photo ionization sensors. Further VOC sensor types include gas chromatography, mass spectrometry, chemiresistors, surface acoustic wave, optical, quartz, field effect transistors, hybrid nanostructures and micro electro mechanical system sensors.
A further type of sensor that can be used for detecting the presence of VOCs is based on color changing indicators. Such sensors are referred to as “colorimetric sensors” and may use an array of thin films of chemically responsive dyes in incrementally varied compositions to produce a measurable color change, for example when exposed to an analyte comprising VOCs and compounds such as acids, alcohols, aldehydes, amines, arenes, ketones, thiols, and the like. The chemoresponsive sensor elements may comprise metalloporphyrins, base indicators, acid indicators, redox dyes, solvatochromic dyes, nucleophilic indicators, or the like. The colorimetric sensor array (“CSA”) response, depending on the type of substrate used, can be either change in intensity of the color or change of the color itself.
Referring now to
In a particular case, the machine vision system measures the response of the sensor array 102 over time by imaging the sensor array 102 exposed to the analyte at intervals and compares the image data to a reference sensor array. In some cases, the sensor array 102 is imaged prior to and after exposure to the analyte. In a particular embodiment, the sensor array 102 is imaged at one hour intervals for a period of 24 hours. The computing module 108 may correlate a difference in intensity of individual sensors in the sensor array 102 to constituents, which may then be further correlated to the parent material responsible for producing the detected VOCs.
The sensor array 102 may comprise a plurality of sensors each configured to respond to the presence of a particular VOC; for example, a first sensor 104-1 configured to respond to the presence of a first VOC in the analyte, and a second sensor 104-2 configured to respond to the presence of a second VOC in the analyte. A single sensor may be configured to respond to multiple VOCs, and multiple sensors may be configured to respond to the same VOC. Sensors in the sensor array 102 may produce a binary or continuous output. Collectively, sensors 104-1, 104-2, and 104-n are referred to as sensors 104, and generically as sensor 104 or individual sensor 104.
In an embodiment, the sensor array 102 is a colorimetric sensor array (“CSA”). The CSA 102 is based on color change as an indicator. The CSA 102 may comprise a plurality of individual sensors 104 arranged in a geometric configuration. In a particular case, the sensors 104 are arranged along a horizontal plane to allow uniform exposure to the analyte. Geometric configurations may include, without limitation, a rectangular or a square grid where individual sensors 104 are uniformly arranged along x- and y-axes on a thin slide, or in a circular petri dish configuration where individual sensors are radially distributed in one or more concentric circles. In a further embodiment, a plurality of colorimetric sensors may be distributed in a three dimensional space to allow exposure of sensors 104 to the analyte while allowing for gravity and buoyancy to naturally separate the constituents of analyte along the vertical axis. The CSA response being measured may depend on the type of substrate used and can include a change in intensity of a color, a change of the color itself, and the like.
The image acquisition device 106 can be any suitable device for acquiring an image of the sensor array 102 and its type is not particularly limited. For example, the image acquisition device 106 may be a point scanner, a flatbed or pass-through scanner, an area scan camera, a line scan camera, or the like. The image acquisition device 106 may be configured to operate in the visible color spectrum or in multi-spectral space. The image acquisition device 106 can be stationary (e.g. mounted to stand at a fixed distance) or portable (e.g. handheld mobile device). In some cases, the image acquisition device 106 may comprise a designated imaging device, such as a single-lens reflex camera used by professional photographers, or may be part of a multipurpose device such as a handheld mobile device or computer tablet. The image acquisition device may be capable of capturing time-lapse images at pre-programmed intervals, or may allow for manual imaging at an operator's request.
The image acquisition device 106 may be capable of embedding exchangeable image file format (EXIF) information in the acquired digital image of the CSA. EXIF data may include: make and model of camera, date, time, resolution, exposure, aperture, F-number, lens-type, focal length, color space, image compression, metering mode, camera flash, and the like. Such information may be used for the purposes of image calibration.
In a particular example, the CSA 102 is a 1.2 cm×2.1 cm patterned CSA spotted with 80 “features,” with each spot being the equivalent of 1.5-2.0 mm in diameter. Of the 80 features, 73 are responsive with the remainder comprising indicators for alignment and control. Each spot includes a fluorescent dye responsive to vapours and possibly radiation. The CSA comprises an array of colored chemical indicators of diverse reactivities embedded in a nanoporous sol-gel matrix. Each indicator in the array sensitively changes color, creating a high dimensional and specific fingerprint allowing identification of the species or mixture present. Because the reactive indicators are highly diverse chemically, a wide range of chemical species can be uniquely detected.
In such an embodiment, the detection event can be observed as a plurality of printed dots changing color. These detection events can be recorded with image acquisition device 106. While kinetic information may be obtained by repeatedly scanning the CSA with the image acquisition device 106, in some cases images may simply be taken before and after exposure of the CSA to the analyte. From the two images of the CSA—the before exposure image and after exposure image—a color difference map can be generated, such as by superimposing and subtracting the two images. The color difference map can be quantified as a high dimensional vector of RGB values. The high dimensional vectors can be used to discriminate between analytes while retaining their chemical resemblances.
In a particular case, the machine vision system captures an image of each sensor color. The image may be quantized in an 8-bit image format with a value between 0 and 255 for each of the three color channels (red, green, and blue). This data may be collected over a regular interval of time; for example, 24 readings of the CSA taken at one hour intervals. This can quickly add up to a large number of data points that can serve as a characteristic digital signature of the analyte. A CSA with 76 sensors imaged at one hour intervals for 24 hours can produce a digital signature with over 5000 data points (i.e. 76 sensors×3 color channels×24 time intervals) for a given analyte.
In an embodiment, as shown in
As described above, use of the sensor array 102 (such as a CSA) can generate a large amount of data. Specialized techniques may be required to perform analysis on data. In some cases, machine learning (“ML”) techniques such as neural networks and statistical pattern recognition may be used to process and analyze the data. In a particular embodiment, ML techniques can be used to develop a reference databank comprising known analytes. The reference databank can be compared with a CSA output from an unknown analyte sample in order to identify the analyte, such as according to a classification process.
In an embodiment, the image processing module 148 performs one or more image processing tasks such as image rotation, alignment, segmentation, white balance adjustment, color correction, blob detection, edge detection, sharpening, smoothing, geometric interpolation, histogram equalization, image normalization, registration, filtering, thresholding, pixel counting, color analysis, denoising, and the like. Image processing tasks can be performed on image data prior to generating color and textural features for the colorimetric sensors.
The image processing module 148 may generate color and textural features from the CSA images, which can be used by the classification module 152 for the purpose of identifying the analyte. “Texture” as used here refers to visual texture rather than physical texture. Textural features can provide information about the spatial arrangement of color or intensities in the image or a selected region of the image. In an embodiment, textural features may include distance and covariance functions such as co-occurrence matrix, angular second moment, contrast, correlation, entropy, Hellinger Distance, the Hausdorff Distance, and the like.
Color features can play an important role in classification. Color features can include global mean, global mode, inner mean, inner mode, and the like. “Global” values of the sensor response can be calculated by computing the color data over all the pixels, while “inner” values can be calculated by computing features using only pixels that are within a certain distance from the centroid. In some cases, extracting color features can require an accurate and robust blob detection method.
In an embodiment, the classification module 152 is configured to implement one or more machine learning (ML) techniques such as principal component analysis, support vector machines, stacked auto encoders, multi-layer perceptrons, recurrent neural networks, deep learning neural networks, and the like, in order to train and develop a classifier for detecting VOCs and identifying the analyte. The ML techniques can be applied to first train one or more classifiers using supervised, unsupervised, or semi-supervised learning techniques. Cross-validation techniques may be applied to reduce prediction error and derive a more accurate estimate of classifier performance. The classifier can be trained to identify the unknown analyte based on image features, such as color and textural features, computed from the CSA images acquired prior to analyte exposure and after analyte exposure.
In a particular embodiment, a recurrent neural network may be used to develop a classifier from time-lapse images of the CSA 102. The time-lapse images provide information regarding changes in color and textural features over time upon exposure to the analyte. The recurrent neural network may comprise architectures including but not limited to a fully recurrent network, a recursive neural network, a Hopfield network, an Elman-Jordan network, a Long short-term memory, a Gated recurrent unit, or the like.
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Referring now to
Based on the detected and interpolated color blobs for the colorimetric sensors from the image processing steps, all the pixels in the blob circle can be acquired. For each color blob, color and textural features can be computed; for example, a mean and mode for each of red, green and blue color pixels. In an embodiment using a CSA having 76 colorimetric sensors, an analyte that is imaged every hour for a 24 hour period can produce a feature vector comprising 24 labeled feature vectors of size 76×3=228 values of color mean differences and 228 values of color mode differences. Thus, in such a case the size of the feature vector for one feature (e.g. color mean) over 24 time lapse images is 24×228=5472. For two features (e.g. color mean and color mode), the size of the feature vector over 24 time lapse images would be 24*228*2=10,944, and so forth. The size of the input feature vector increases dramatically with an increase in the number of time lapse images and number of features (whether color, textural, or other). Techniques such as principal component analysis can be performed to reduce the dimensionality of data, which may in turn help reduce variance of a predictor or classifier, such as a support vector machine.
Referring now to
Referring now to
Referring now to
Outputs from the image processing techniques can be provided to and used by the computing module 150 in further processing and analysis. For example, machine learning techniques may be applied in order to perform VOC detection and analyte identification.
In an embodiment, the detection of VOCs in an analyte or the identification of the analyte itself and other processing of imaging data for evaluation purposes can be based on computational modules. Computational modules can be implemented using any computational paradigm capable of performing data analysis based on various methods such as regression, classification and others. In some variations, the computational modules can be learning based. One learning based computational paradigm capable of performing such methods may be a neural network. Neural networks may include Restricted Boltzmann Machines, Deep Belief Networks, and Deep Boltzmann Machines. Accordingly, a neural network can be used to identify an analyte based on component VOCs. Thus, feature data such as color and textural feature data, as well as relevant data from databases and other services, can be provided to a neural network, which can perform analyte identification based on classification/regression or similar methods.
In embodiments, analysis and classification by the computing module 108 may be implemented by providing input data to a neural network, such as a feed-forward neural network, for generating at least one output. The neural networks may have a plurality of processing nodes, including a multi-variable input layer having a plurality of input nodes, at least one hidden layer of nodes, and an output layer having at least one output node. During operation of a neural network, each of the nodes in the hidden layer applies an activation/transfer function and a weight to any input arriving at that node (from the input layer or from another layer of the hidden layer), and the node may provide an output to other nodes (of a subsequent hidden layer or to the output layer). The neural network may be configured to perform a regression analysis providing a continuous output, or a classification analysis to classify data. The neural networks may be trained using supervised or unsupervised learning techniques, as described below. According to a supervised learning technique, a training dataset is provided at the input layer in conjunction with a set of known output values at the output layer. During a training stage, the neural network may process the training dataset. It is intended that the neural network learn how to provide an output for new input data by generalizing the information it learns in the training stage from the training data. Training may be effected by back propagating the error to determine weights of the nodes of the hidden layers to minimize the error. Once trained, or optionally during training, test (verification) data can be provided to the neural network to provide an output. A neural network may thus cross-correlate inputs provided to the input layer in order to provide at least one output at the output layer. Preferably, the output provided by a neural network in each embodiment will be close to a desired output for a given input, such that the neural network satisfactorily processes the input data.
The term “classification” as used herein should be understood in a larger context than simply to denote supervised learning. By classification process we convey: supervised learning, unsupervised learning, semi-supervised learning, active/groundtruther learning, reinforcement learning and anomaly detection. Classification may be multi-valued and probabilistic in that several class labels may be identified as a decision result; each of these responses may be associated with an accuracy confidence level. Such multi-valued outputs may result from the use of ensembles of same or different types of machine learning algorithms trained on different subsets of training data samples. There are various ways to aggregate the class label outputs from an ensemble of classifiers; majority voting is one method.
In some embodiments, an ensemble of classifiers may be used, such as multiple support vector machines or other classifiers running simultaneously. The classification ensemble may be homogeneous or heterogeneous. A homogeneous ensemble comprises a plurality of classifiers of the same machine learning type (for example, multiple support vector machines). Each classifier in a homogeneous ensemble may have different parameter values and may be trained using a distinct subset of the samples in the training set. A heterogeneous ensemble comprises a plurality of classifiers belonging to a variety of machine learning algorithms; for example, a deep neural network, a K-means clustering, and an SVM. Classifiers in heterogeneous ensembles may be trained on the same training data or on distinct subsets of the training data. If a multiplicity of a machine learning algorithms exists in a heterogeneous ensemble, each instance of the multiplicity may be trained on some samples unique only to that instance.
In some variations, the neural network can operate in at least two modes. In a first mode, a training mode, the neural network can be trained (i.e. learn) based on known images. The training typically involves modifications to the weights and biases of the neural network, based on training algorithms (backpropagation) that improve its detection capabilities. In a second mode, a normal mode, the neural network can be used to detect VOCs in the analyte using an image of an exposed sensor array. In variations, some neural networks can operate in training and normal modes simultaneously, thereby both detecting the presence of VOCs, and training the network based on the detection effort performed at the same time to improve its detection capabilities. In variations, training data and other data used for performing detection services may be obtained from other services such as databases or other storage services. Some computational paradigms used, such as neural networks, involve massively parallel computations. In some implementations, the efficiency of the computational modules implementing such paradigms can be significantly increased by implementing them on computing hardware involving a large number of processors, such as graphical processing units.
In variations, VOC detection using a neural network or clustering mechanism can be an ongoing process. For example, in some implementations, the computing module can be a local computing module and provide results to a remote computing module. The remote computing module can include appropriate learning mechanisms to update a training model based on the newly received signals. For example, the remote computing module can be a neural network based system implemented using various application programming interfaces APIs and can be a distributed system. The APIs included can be workflow APIs, match engine APIs, and signal parser APIs, allowing the remote computing module to both update the network and determine whether a VOC is present in the analyte.
Embodiments of the systems and methods of the present disclosure may implement groundtruthing to ensure classification result accuracy according to an active learning technique. Specifically, results from classification models may be rated with a confidence score, and high uncertainty classification results can be pushed to a groundtruther to verify classification accuracy. Optionally, classification outputs can periodically be provided to groundtruthers to ensure accuracy. In some implementations, a determination by the system indicative of the presence of a VOC may result in generating a request for human groundtruthing of the detection signal or the sensor array or sensor array image.
Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto. The entire disclosures of all references recited above are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62572600 | Oct 2017 | US |
