Patent Application
20220303516
This application claims priority of Taiwanese Patent Application No. 110109760, filed on Mar. 18, 2021.
The disclosure relates to a method of air pollution estimation based on spectral image processing.
Air pollution has become an important issue in modern day lives, with the advancement of technology, the manufacturing industry, and associated emissions of particles. Typically, a number of aerosol particles (e.g., solid particles, gas particles, liquid particles, etc.) suspended in the air may cause air pollution. These particles are mostly small in size and cannot be directly perceived by the human eye; however, the potential harmful effect on human beings exposed to these particles is immense. According to statistical data from the World Health Organization (WHO), about 7 million people die each year from diseases (such as respiratory tract infection, chronic obstructive pulmonary disease (COPD), heart diseases, etc.) caused by air pollution. As a result, detection of the severity of air pollution becomes desirable.
Conventionally, detection of the severity of air pollution may be done by using instruments (e.g., a tapered element oscillating microbalance (TEOM), a high volume air sampler, a low volume air sampler, etc.) to collect samples of aerosol particles suspended in the air. In one example, air is continuously pumped into an inlet of the instrument for one day. The instrument is configured to filter out the larger particles such that the particles with a diameter equal to or smaller than 2.5 μm (also known as fine particles or PM2.5) are collected onto a filter (e.g., a filter paper). By measuring a weight of the filter before the operations and a weight of the filter after the day of operations, a difference therebetween may be calculated and serve as a net weight of the fine particles collected throughout the day. By dividing the net weight by the volume of the air pumped into the instrument throughout the day, an average concentration of fine particles during the day can be obtained.
It is noted that the filter paper needs to be carefully pre-processed and weighed so as to determine a precise weight thereof. Also, the instrument, especially the inlet, needs to be carefully calibrated to prevent articles other than the fine articles from ending up on the filter paper.
An article by Z. He, X. Ye, K. Gu & J. Qiao, “Learn to predict PM2.5 concentration with image contrast-sensitive features”, in 2018 37th Chinese Control Conference (CCC), 2018: IEEE, pp. 4102-4106 discloses a model for estimating the concentration of the fine particles using image contrast-sensitive features. In this document, a recurrent fuzzy neural network model is established to extract features from colored images captured by, for example, a camera, so as to estimate the concentration of the fine particles. Parameters of the recurrent fuzzy neural network model may be trained using a gradient descent algorithm. with self-adaptive learning rate.
One object of the disclosure is to provide a method that is capable of implementing a relatively more accurate estimation of a severity of air pollution.
According to the disclosure, the method includes is implemented using a processor of a computer device and includes:
A) generating a spectral image based on an original color image of an environment under test using a spectral transformation matrix;
B) supplying the spectral image as an input into an estimating model for air pollution estimation; and
C) from the estimating model, obtaining an estimation result indicating a degree of air pollution of the environment under test.
Another object of the disclosure is to provide a method for obtaining a spectral transformation matrix to be used in performing the above-mentioned air pollution estimation.
According to the disclosure, the method is implemented using a processor of a computer device. The computer device stores a reference color image of a reference object and a reflection spectral dataset associated with the reference object. The method includes:
performing a converting operation to convert the reference color image and the reflection spectral dataset to a CIE 1931 XYZ color space created by the International Commission on Illumination (CIE) in 1931, so as to obtain a converted reference image and a converted spectral dataset corresponding to the reference color image and the reflection spectral dataset, respectively;
obtaining a parameter matrix from the converted reference image;
obtaining an adjustment matrix from the converted spectral dataset and the parameter matrix;
obtaining a corrected image based on the parameter matrix and the adjustment matrix; and
obtaining the spectral transformation matrix from the reflection spectral dataset and the corrected image.
Other features and advantages of the disclosure will become apparent in the following detailed description of the embodiments with reference to the accompanying drawings, of which:
Before the disclosure is described in greater detail, it should be noted that where considered appropriate, reference numerals or terminal portions of reference numerals have been repeated among the figures to indicate corresponding or analogous elements, which may optionally have similar characteristics.
Throughout the disclosure, the term “coupled to” may refer to a direct connection among a plurality of electrical apparatus/devices/equipments via an electrically conductive material (e.g., an electrical wire), or an indirect connection between two electrical apparatus/devices/equipments via another one or more apparatus/devices/equipments, or wireless communication.
The processor 12 may include, but not limited to, a single core processor, a multi-core processor, a dual-core mobile processor, a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), etc.
The communication unit 13 is connected to the processor 12, and may include at least one of a radio-frequency integrated circuit (RFIC), a short-range wireless communication module supporting a short-range wireless communication network using a wireless technology of Bluetooth® and/or Wi-Fi, etc., or a mobile communication module supporting telecommunication using Long-Term Evolution (LTE), the third generation (3G) and/or fifth generation (5G) of wireless mobile telecommunications technology, and/or the like.
The data storage 11 is connected to the processor 12, and may be embodied using computer-readable storage medium such as hard disk drive(s), random access memory (RAM), read only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc.
In this embodiment, the data storage 11 stores a software application program, a plurality of training color images of a specific environment, a plurality of severity tags associated respectively with the training color images and each indicating a degree of air pollution, a reference color image of a reference object, and a reflection spectral dataset associated with the reference object. The software application program includes instructions that, when executed by the processor 12, cause the processor 12 to perform a number of operations as described in the subsequent paragraphs.
The training color images may be captured using a camera, and may each be in the form of an 8-bit color image. The camera may be mounted on a drone for capturing images of nearby environment. Specifically, each of the training color images is an aerial view. The reference object may be X-Rite ColorChecker® Classic, which includes twenty-four color squares with twenty-four common natural colors, respectively. The reference color image may be captured using a camera, and may be in the form of an 8-bit color image expressed using a standard red, green, blue (sRGB) color space. In some embodiments, the reference color image and the training color images are captured using the same camera. The reflection spectral dataset may include spectral data (spectrum) of light reflected by the reference object, measured by, for example, a spectrometer, and correspond to wavelengths ranging from 380 to 780 nm and have a spectral resolution of 1 nm. In the reflection spectral dataset, the wavelengths correspond to a plurality of spectral values (i.e., intensity values), respectively. The severity tags may include, for example, three types of tags of “Good”, “Moderate” and “Severe”, each of which corresponds to a specific range of concentration of aerosol particles with a diameter equal to or smaller than 2.5 μm (also known as fine particles or PM2.5). Each of the three types of tags may be also associated with a specific color (e.g., green for “Good”, yellow for “Moderate” and red for “Severe”). It is noted that in some embodiments, the severity tags may include more than three types of tags.
In use, it is desired to estimate a severity of air pollution of an environment under test using a color image of the environment under test captured using a camera. In some embodiments of the disclosure, a method of air pollution estimation based on spectral image processing may include a matrix calculating process, a model constructing process, and an air pollution estimating process.
In step 21, the processor 12 performs a converting operation on the reference color image and the reflection spectral dataset to convert the same to a CIE 1931 XYZ color space created by the International Commission on Illumination (CIE) in 1931, so as to obtain a converted reference image and a converted spectral dataset corresponding to the reference color image and the reflection spectral dataset, respectively.
Specifically, the converting operation includes, with respect to the reference color image, obtaining the converted reference image using the following equations:
where XC, YC end ZC represent an X value, a Y value and a Z value of a pixel of the converted reference image in the CIE 1931 XYZ color space, RsRGB, GsRGB and BsRGB represent a red value, a green value and a blue value of a pixel of the reference color image in the sRGB color space corresponding to the pixel of the converted reference image, XCW, YCW and ZCW represent a red value, a green value and a blue value of a white point defining the white color in the sRGB color space that is captured under the CIE standard illuminant D65), XSW, YSW and ZSW represent a white point for an environmental illuminant under which the reference color image was captured, and [MA] represents a chromatic adaptation matrix. The white point is a set of tristimulus values or a set of chromaticity coordinates.
It is noted that, since the environmental illuminant used for capturing the reference color image may be one other than the standard illuminant in the sRGB color space, the chromatic adaptation matrix [MA] is employed for converting the white point for the standard illuminant to a corresponding white point for the environmental illuminant of the reference color image.
The converting operation also includes, with respect to the reflection spectral dataset, obtaining the converted spectral dataset using the following equations:
X
S
=k∫
380 nm
780 nm
S(λ)R(λ)
Y
S
=k∫
380 nm
780 nm
S(λ)R(λ)
Z
S
=k∫
380 nm
780 nm
S(λ)R(λ)
k=100/∫280 nm780 nmS(λ)
where, for each wavelength λ of the spectrum of the reflection spectral dataset (having a range between 380 to 780 nanometers), XS, YS and ZS represent an X value, a Y value and a Z value of the converted spectral dataset in the CIE 1931 XYZ color space for the spectral value corresponding to the wavelength λ of the spectrum of the reflection spectral dataset, S(λ) represents a spectral value corresponding to the wavelength λ in a spectrum of the environmental illuminant under which the reference color image was captured, R(λ) represents the spectral value corresponding to the wavelength λ of the spectrum of the reflection spectral dataset of the reference object, and
It is noted that the converting operation as described above is performed in a pixel-to-pixel basis.
In step 22, the processor 12 obtains a parameter matrix from the converted reference image that is obtained in step 21.
Specifically, the parameter matrix is used to correct errors attributed to a number of factors that are associated with the camera used to capture the reference color image. In this embodiment, the factors involved include a non-linear response of the camera, a dark current associated with the camera, a deviation of a color filter of the camera, and a color shift of the camera (e.g., white balance (WB)). One or more matrices may be used to express correction parameters for correcting one or more of the above mentioned factors. Afterward, the parameter matrix may be obtained based on the one or more matrices for the above mentioned factors.
In this embodiment, a first matrix for the non-linear response of the camera may be expressed as)
V
Non-linear=[XC3 YC3 ZC3 XC2 YC2 YC2 XC YC ZC 1]T.
A second matrix for the dark current associated with the camera may be expressed as VDark=[a], where a is a constant indicating the dark current, which is typically constant.
A third matrix for the deviation of the color filter of the camera and the color shift of the camera may be expressed as
V
Color=[XCYCZC XCYC XCZC YCZC XC YC ZC]T,
where, since the reference color image has been converted to the CIE 1931 XYZ color space, the X, Y and Z values XC, YC and ZC of the pixel of the converted reference image are considered.
Using the above first to third matrixes, the parameter matrix is represented by:
V=[XC3 YC3 ZC3 XC2YC XC2ZC YC2ZC XCYC2 XCZC2 YCZC2 XCYCZC XC2 YC2 YC2 XCYC XCZC YCZC XC YC ZC α]T.
In step 23, the processor 12 obtains an adjustment matrix based on the converted spectral dataset and the parameter matrix. Specifically, the adjustment matrix [C] is obtained by performing multiple regression analysis based on the following equation:
C=[XYZSpetrum]×pinv([V])
where [XYZspectrum] is a matrix containing the X, Y and Z values of the converted spectral dataset in the CIE 1931 XYZ color space, [V] is the parameter matrix, and pinv([V]) represents an inverse matrix of the parameter matrix.
In step 24, the processor 12 obtains a corrected image based on the parameter matrix and the adjustment matrix. Specifically, the processor 12 obtains the corrected image using the following equation:
[XYZCorrect]=[C]×[V]
where [XYZCorrect] is a matrix containing X values, Y values and Z values of all pixels of the corrected image in the CIE 1931 XYZ color space. Using the above operation, the pixel values of the pixels that constitute the corrected image may be obtained, and the processor 12 is programmed to generate the corrected image accordingly.
In step 25, the processor 12 obtains a spectral transformation matrix from the reflection spectral dataset and the corrected image. Specifically,
In sub-step 251, the processor 12 performs a principal components analysis (PCA) on the reflection spectral dataset, so as to obtain a plurality of principal components, and a principal component score and a principal component eigenvalue for each of the principal components. In this embodiment, twelve principal component scores and twelve principal component eigenvalues of twelve principal components are obtained, as these principal component scores and principal component eigenvalues are capable of explaining 99.9964854% of variance within the reflection spectral dataset, but other numbers of principal component scores and principal component eigenvalues may be employed in other embodiments.
In sub-step 252, the processor 12 performs multiple regression analysis on the plurality of principal component scores and the corrected image, so as to obtain the spectral transformation matrix.
Specifically, the operation of sub-step 252 includes using the following equation to obtain the spectral transformation matrix M:
M=[Score]×pinv([VColor′])
where [Score] is a matrix that contains the plurality of principal component scores,
[VColor′]=[XC′YC′ZC′ XC′YC′ XC′ZC′ YC′ZC′ XC′ YC′ ZC′]T, and X′C, Y′C and Z′C represent an X value, a Y value and a Z value of each pixel of the corrected image in the CIE 1931 XYZ color space.
With the spectral transformation matrix, it is then possible to construct a neural network model for performing the estimation of the severity of air pollution of the environment under test.
In step 31, the processor 12 performs a transformation operation on each of the training color images using the spectral transformation matrix, so as to obtain a plurality of transformed training images that correspond with the training color images, respectively. The training color images may be pre-stored in the data storage 11, or may be obtained from an external source via the communication unit 13.
In step 32, the processor 12 performs a dimension reduction operation on each of the transformed training images, so as to obtain a plurality of dimension-reduced datasets associated with the transformed training images, respectively. In this embodiment, the dimension reduction operation may be done using a three-dimensional (3D) convolutional autoencoder or the PCA, but is not limited as such.
In step 33, the processor 12 uses the dimension-reduced datasets and the severity tags that correspond respectively with the training color images to train a neural network of a machine learning algorithm, so as to construct an estimating model for performing the air pollution estimation of the environment under test. Specifically, to train the neural network, the dimension-reduced datasets serve as inputs to the neural network, and the severity tags serve as known outputs of the neural network. That is to say, for each of the training color images of the specific environment, a concentration of fine aerosol particles of the specific environment is known (e.g., from official data of air quality monitoring), and therefore a type of the severity tag corresponding to the training color image is known and will be supplied to the neural network of the machine learning algorithm, for the neural network to “learn” about a relationship between the dimension-reduced datasets associated with the training color images and the associated severity of air pollution.
In this embodiment, the neural network of the machine learning algorithm may be a convolutional neural network (CNN) with a VGG16 (also known as OxfordNet) architecture. After a number of training operations, the estimating model can be used for performing air pollution estimation on the environment under test using an image of the environment under test. It should be noted that, in the case that the machine learning algorithm is with the VGG16 architecture, each of the training color images may be adjusted to a size of 224×224 pixels to fit a size of the first layer of the VGG16 architecture.
In step 41, the processor 12 generates a spectral image based on the original color image of the environment under test using the spectral transformation matrix. Specifically,
In sub-step 411, the processor 12 converts the original color image to the CIE 1931 XYZ color space, so as to obtain a converted original image. The operations of sub-step 411 may be done in a manner similar to that of step 21, and details thereof are omitted herein for the sake of brevity.
In sub-step 412, the processor 12 obtains the spectral image using the following equation:
[SSpectrum]=[EV][M][VColor″]
where [SSpectrum] is a matrix containing an X value, a Y value and a Z value of each of the pixels of the spectral image in the CIE 1931 XYZ color space, [M] is the spectral transformation matrix, [EV] represents the principal component eigenvalues, and
[VColor″]=[XTYTZT XTYT XTZT YTZT XT YT ZT]T
where XT, YT and ZT represent an X value, a Y value and a Z value of each pixel of the converted original image in the CIE 1931 XYZ color space, respectively.
It is noted that, the operation of step 41 may be done in a manner similar to that of step 31, and details thereof are omitted herein for the sake of brevity. After step 41, the spectral image that corresponds with the original image and that is in the CIE 1931 XYZ color space is generated.
In step 42, the processor 12 supplies the spectral image as an input into the estimating model for air pollution estimation.
In step 43, the processor 12 obtains an estimation result from the estimating model indicating a degree of air pollution of the environment under test. Specifically, the estimating model may detect the colors of the spectral image, so as to assign to the spectral image a severity tag selected from the three types of tags. As such, the assigned severity tag is outputted by the estimating model for notifying a user of an estimated air pollution severity of the environment under test. For example, a color associated with the type of the assigned severity tag may be displayed on a display screen (not shown).
To sum up, embodiments of the disclosure provide a method for air pollution estimation based on spectral image processing. In the method, a matrix calculating process is implemented to extract the accurate spectral information (e.g., the principle component scores, the principle component eigenvalues, etc.) from the reference object in the CIE 1931 XYZ color space. Also, various parameters that may cause errors in subsequent estimations are also adjusted in the spectral transformation matrix.
Then, using the spectral transformation matrix, the training color images may be transformed before being used to train the neural network of machine learning to construct the estimating model for actually performing the air pollution estimation.
Afterward, the air pollution estimating process is performed by first transforming the original color image of the environment under test to the spectral image using the spectral transformation matrix, and then supplying the spectral image to the estimating model, so as to obtain the estimation result from the estimating model where the estimating result indicates a degree of air pollution of the environment under test. As such, using an image of a specific environment, the air pollution estimation process of this disclosure may be performed with more accuracy than the prior art.
In the description above, for the purposes of explanation, numerous specific details have been set forth in order to provide a thorough understanding of the embodiment. It will be apparent, however, to one skilled in the art, that one or more other embodiments may be practiced without some of these specific details. It should also be appreciated that reference throughout this specification to “one embodiment,” “an embodiment,” an embodiment with an indication of an ordinal number and so forth means that a particular feature, structure, or characteristic may be included in the practice of the disclosure. It should be further appreciated that in the description, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of various inventive aspects, and that one or more features or specific details from one embodiment may be practiced together with one or more features or specific details from another embodiment, where appropriate, in the practice of the disclosure.
While the disclosure has been described in connection with what are considered the exemplary embodiments, it is understood that this disclosure is not limited to the disclosed embodiments but is intended to cover various arrangements included within the spirit and scope of the broadest interpretation so as to encompass all such modifications and equivalent arrangements.
| Number | Date | Country | Kind |
|---|---|---|---|
| 110109760 | Mar 2021 | TW | national |
