Patent Application
20210398281
Farmers are increasingly using agricultural imaging of outdoor crops in an effort to improve crop yields. Conventional agricultural imaging techniques can facilitate easier inspection of outdoor crops, greater precision of crop monitoring, and earlier detection of crop problems, such as certain nutrient deficiencies, insect infestations, and diseases. Satellites, aircraft, or unmanned aerial vehicles (UAVs) are used to capture hyperspectral or multispectral aerial images of the crops. These aerial images can be processed using vegetation indices (VIs) to determine certain crop features.
A VI is a spectral transformation of two or more spectral reflectance measurement bands that takes advantage of differences in light reflectance from vegetation in different spectral bands. VIs may facilitate indicating an amount of vegetation, distinguishing between soil and vegetation, or reducing atmospheric and topographic effects in images of vegetation. VIs can be correlated with various crop characteristics, such as leaf area index (LAI), percent green cover, chlorophyll content, green biomass, and absorbed photosynthetically active radiation (APAR). The Normalized Difference Vegetation Index (NDVI) is a common VI that compares visible red and near-infrared (NIR) spectral reflectance bands. Other common VIs include the Photochemical Reflectance Index (PRI), the Difference Vegetation Index (DVI), the Ratio Vegetation Index (RVI), and the Crop Water Stress Index (CWSI). Images that are generated and/or processed based on one or more VIs can show changes in crops weeks before the changes are visible to the naked eye. Such insights from images can be used to improve resource use efficiency; protect crops from certain diseases, pests, and water stress; and improve crop yield.
In conventional agricultural imaging techniques, multiple parameters effect the efficacy and utility of images of crops; examples of such parameters include pixel resolution (size of the pixels), image bandwidth (spectral range of wavelengths being imaged), radiometry resolution (range of relative radiation reflectance values per pixel), and positional accuracy. All of these parameters impact the usefulness of the image for crop monitoring. In particular, agricultural imaging equipment that provides sufficient imaging resolution and accuracy generally translates to increased costs to the farmer. In some instances, the cost of effective agricultural imaging for a given farmer can offset any increased profits due to better yields, leaving the potential agricultural advantages of such imaging out of reach for some farmers.
Controlled Environment Horticulture (CEH) (also referred to more generally as controlled environment agriculture or CEA) is the process of growing plants in a controlled environment where various environmental parameters are monitored and adjusted to improve the quality and yield of the plants grown. Compared to conventional approaches of plant cultivation, CEH may enable year-round production of plants, insensitivity to variable weather conditions, reduce pests and diseases, and reduce the number of resources consumed on a per plant basis. A controlled horticultural environment is typically enclosed, at least in part, by a building structure such as a greenhouse, a grow room, or a cover for a portion of a field in order to provide some degree of control over environmental conditions. Additional control systems may be deployed in CEH to adjust various environmental parameters including lighting, temperature, humidity, nutrient levels, and carbon dioxide (CO2) concentrations. For example, one or more artificial lighting systems are often used in such controlled horticultural environments to supplement and/or replace natural sunlight that may be obstructed by the building structure or insufficient during certain periods of the year (e.g., winter months).
There have been multiple attempts to adapt agricultural imaging systems for CEH. For example, in the article by Story and Kacira, “Design and Implementation of a Computer Vision-Guided Greenhouse Crop Diagnostics System,” Machine Vision and Applications, Vol. 26 (2015), pp. 495-506, the authors describe a canopy imaging system with robotic positioning to move the imaging system around a CEH growth environment. The imaging system determined crop features, such as color, texture, morphology, and temperature under greenhouse field lighting conditions, using three cameras for visible, near infrared (NIR), and thermal imaging, respectively. Robotic positioning moved the three-camera housing within an xy-coordinate system above the crop canopy.
As another example, international patent publication WO 2017/192566 describes a horticultural lighting system with one or more modular devices, such as a hyperspectral, stereoscopic, or infrared camera, installed into a receptacle in a lighting device. The modular devices can be used to validate light levels and spectrum delivered to growth environments in order to identify light intensity decay and spectrum shift over time.
As a third example, U.S. Pat. No. 8,850,742 describes a lighting system used to modulate and control growth and attributes of greenhouse crops. The system includes arrays of high power LEDS having different ranges of wavelengths to irradiate one or more plants in the CEH, as well as sensors or cameras with specific color filters to sense reflected or emitted light from the crops. This system uses the data collected from the sensors or cameras to predict response and performance of plants to various growing conditions and modulates the lighting produced by the lighting system based on the data.
In addition to imaging, more generally a variety of environmental sensing equipment available from multiple manufacturers has been conventionally employed to monitor different conditions and aspects of crops and their environs. Like agricultural imaging, incorporating multiple sensing modalities in CEH may provide greater insight and control on environmental parameters pertinent to plant development, crop yield, and crop quality. Various types of conventional sensors deployed in CEH, however, often are installed, connected, and controlled individually and/or in an uncoordinated manner (e.g., particularly if each sensor type from a given manufacturer has a proprietary connection mechanism). This in turn may outweigh the potential benefits of increased monitoring of crops via diverse sensing; in particular, increasing the number of diverse sensors unduly increases the complexity of the sensing system and thereby may pose a greater time, cost and/or other resource burden on farmers.
As noted above, aerial agricultural imaging is limited to outdoor agricultural environments. Accordingly, there are multiple considerations for adapting agricultural imaging to Controlled Environment Horticulture (CEH), given that crops are not accessible for aerial imaging (e.g., using satellites, planes, or UAVs). Additionally, one or more artificial lighting systems often are used in CEH to supplement and/or replace natural sunlight that may be obstructed by the building structure or insufficient during certain periods of the year (e.g., winter months).
The Inventors have recognized and appreciated that previous efforts to adapt conventional agricultural imaging techniques to CEH have had some shortcomings. For example, some previous efforts, such as those disclosed in WO 2017/192566 and Story and Kacira, relied on illumination from the growing environment (e.g., either artificial growing lights or the sun) to provide sufficient light for imaging. However, the Inventors have recognized and appreciated the benefits of augmenting an artificial lighting system for CEH with an imaging system that includes its own irradiation devices; such an imaging system allows crops to be irradiated with specific narrow spectral bands to sense and image crop conditions that may be particularly represented in one or more of these narrow spectral bands. The Inventors have also recognized the importance of measuring spectral characteristics of crops in situ without necessarily affecting morphological changes in the crops due to irradiation from an imaging system.
To this end, the Inventors have appreciated some of the limitations of imaging techniques such as those disclosed techniques in U.S. Pat. No. 8,850,742, which employed simultaneous irradiation of plants using several high power narrow spectral band irradiators. However, by simultaneously irradiating plants with several relatively high power and narrow spectral bands, the techniques employed in this patent mask or make it difficult to isolate a particular characteristic of the irradiated crops at any one narrow spectral band. When considered together with the relatively low-resolution cameras employed, the disclosed techniques in U.S. Pat. No. 8,850,742 compromise accuracy and reliability. Additionally, the disclosed high-power irradiating techniques intentionally modified biochemical attributes of the plants, and hence were not oriented to measuring various aspects of plants in situ in the nominal conditions of the controlled environment.
In view of the foregoing, the present disclosure is directed generally to multisensory imaging methods and apparatus that involve one or both of multispectral imaging and integrated sensing to provide a fuller compliment of information regarding crops in CEH, from an entire grow space, to some smaller portion of a grow space, down to individual plants, leaves, buds, or other plant constituents. In example implementations discussed herein, comprehensive multisensory imaging may be realized in relatively large “fields of view” in a given grow space, (in which the resolution of data captured in a given image may be on the scale of multiple plants or groups of plants or a larger portion of a grow space), as well as relatively smaller fields of view (in which the resolution of data captured by a given image may be on the scale of a single plant or portion of a plant). In one salient aspect, various data constituting an image, acquired from virtually any size field of view or any image resolution, is indexed as respective pixels representing points in a one-dimensional (1D), two-dimensional (2D) or three-dimensional (3D) arrangement of sensing nodes in some portion (or all of) a grow space.
The various concepts disclosed herein constitute significant improvements in horticultural imaging that reduce cost, improve access of agricultural imaging to farmers, and improve image quality and the quantum of information that can be derived from the images. These concepts also constitute significant improvements in horticultural sensing more generally that reduce cost, improve access to farmers, and enhance the quantum of information that can be derived from sensors deployed in CED. Furthermore, in some example implementations, the disclosed concepts integrate agricultural sensing and imaging together for CEH, and further integrate sensing and imaging concepts with artificial lighting and environmental conditioning (e.g., HVAC) for CEH to provide holistic control and monitoring solutions.
For example, one implementation disclosed herein relates to a multispectral imaging system that is deployed in combination with a fluid-cooled light emitting diode (LED)-based lighting fixture (also referred to hereafter as a “lighting fixture” or “illuminator”). The multispectral imaging system may provide finite spectra sensing to measure narrowband spectra (e.g., about 2 nm to about 40 nm). The finite spectra sensing capabilities provided by the multispectral imaging system may enable the characterization of various aspects of CEH crops including, but not limited to deep plant phenotyping, plant-environment interactions, genotype-phenotype relations, growth rate correlations, imaging, and analysis of plants in pots, containers, and/or ground soil. Furthermore, the multispectral imaging system is sufficiently compact for deployment and use at length scales less than 100 meters (unlike previous imaging systems deployed on aircraft, i.e., at length scales greater than 1000 meters).
In one aspect, the multispectral imaging system may be integrated with a lighting fixture or a separate module (or accessory) connected to the lighting fixture via a wired or wireless connection. The multispectral imaging system may include imagers/sensors to acquire imagery and spectra on the agricultural environment. The imagers/sensors may be configured to acquire imagery and/or spectra over a broad wavelength range (e.g., ultraviolet to long wavelength infrared). In some implementations, the multispectral imaging system may include a first imager/camera to acquire imagery from the UV to short wavelength infrared (SWIR) regimes and a second imager/camera to acquire imagery in the long wavelength infrared (LWIR) regime. In some implementations, the second imager/camera may continually acquire LWIR imagery while the first imager/camera is periodically activated in combination with an illumination source (e.g., the onboard LED elements) to acquire UV-SWIR imagery.
In some implementations, the multispectral imaging system may include one or more irradiation sources (e.g., LED elements) to illuminate the agricultural environment with radiation at different wavelengths. In this manner, the imagers/sensors may acquire imagery corresponding to the radiation at each respective wavelength, which may then be overlaid to form a multispectral image. In some implementations, the multispectral imaging system may instead use LED light sources integrated into the lighting fixture to illuminate the environment at one or more wavelengths. The multispectral imaging system may also include one or more filters (e.g., a filter wheel) to selectively acquire imagery/spectra data at a particular wavelength if the illuminator (e.g., the LED light sources, the onboard LED elements) illuminate the environment with radiation at multiple wavelengths.
In one exemplary method, the multispectral imaging system may acquire imagery/spectra of the agricultural environment using the following steps: (1) turning on a first LED element in the multispectral imaging system to irradiate the agricultural environment with radiation at a first wavelength, (2) acquiring imagery/spectra of the environment using the imagers/sensors in the multispectral imaging system, and (3) turning off the first LED element. This method may be repeated for additional LED elements in the multispectral imaging system using, for example, the following steps: (4) turning on a second LED element in the multispectral imaging system to irradiate the agricultural environment with radiation at a second wavelength, (5) acquiring imagery/spectra of the environment using the imagers/sensors in the multispectral imaging system, and (6) turning off the second LED element.
In some implementations, the multispectral imaging system may acquire imagery/spectra of the environment while no other illumination sources are active. For example, the lighting fixture may include LED light sources that provide photosynthetically active radiation (PAR). The LED light sources may be turned off when the multispectral imaging system acquires imagery/spectra of the environment. Said in another way, the multispectral imaging system may be configured to irradiate a dark environment with a series of flashes (e.g., brief pulses of radiation) to acquire imagery/spectra at several wavelengths.
The multispectral imaging system may also include supplementary LED elements. In some implementations, the supplementary LED elements may be used to modify the plants and/or their surroundings. For example, one or more of the supplementary LED elements may emit UV radiation with sufficient brightness (and/or intensity) to repel pests or to reduce the growth of mildew (e.g., using 275 nm wavelength radiation). In another example, one or more of the supplementary LED elements may alter the morphology and/or the photoperiod of the plants (e.g., using 730 nm wavelength radiation). It should be appreciated that the light treatment functionality provided by the supplementary LED elements may also be provided by the LED light sources that illuminate the plants with PAR.
In some implementations, the multispectral imaging system may be paired with another irradiator that provides radiation to the environment. The irradiator may provide radiation that covers the UV, visible, near infrared (NIR), and/or SWIR regimes. In some implementations, the lighting fixture may be paired with another imager/sensor. The imager/sensor may be configured to acquire imagery/spectra covering the UV, visible, NIR, SWIR, and/or LWIR regimes. In general, the imager/sensor may acquire 2D imagery and/or 3D imagery (e.g., Lidar, a pan-tilt-zoom (PTZ) camera) of the environment. The imager/sensor may also have a field of view that ranges between a portion of the environment that includes one or more plants to an entire room of the environment. Note the environment may contain multiple rooms.
In some implementations, the multispectral imaging system may be calibrated using various calibration sources disposed in the environment. In one example, a phantom may be used to calibrate imagery/spectra acquired between the UV and SWIR regimes. The phantom may be an object with known optical properties (e.g., emissivity, absorptivity, reflectivity) with various shapes (e.g., a sphere, a polyhedron, a plant, an animal). One or more phantoms may be placed within the field of view of the imager/sensors to calibrate the magnitude and wavelength of radiation detected by the imager/sensors. In another example, a blackbody reference may be used to calibrate thermal imagery/spectra acquired in the LWIR regime. The blackbody reference may be an object that includes a heater and a temperature sensor. The blackbody reference may be used to calibrate the temperature value in a heatmap acquire by a LWIR imager/sensor.
In various implementations, the imaging system disclosed herein may include one or more communication and/or auxiliary power ports, for example, to provide auxiliary DC power to one or more auxiliary devices coupled to the port(s). Example of such ports include, but are not limited to, one or more Power over Ethernet (PoE) ports and/or one or more Universal Serial Bus (USB) ports to communicatively couple multiple lighting fixtures together and/or support operation of one or more auxiliary devices (e.g., sensors, actuators, or other external electronic devices). Examples of various sensors that may be coupled to one or more imaging systems via one or more of the PoE or USB ports include, but is not limited to, air temperature sensors, near-infrared (NIR) leaf moisture sensors, hyperspectral cameras, finite spectral cameras, IR leaf temperature sensors, relative humidity sensors, and carbon dioxide sensors. Other examples of auxiliary devices that may be coupled to one or more imaging systems via PoE or USB ports include, but are not limited to, one or more fans, security cameras, smart phones, and multi-spectral cameras (e.g., to analyze soil moisture, nutrient content, leaves of the plants). In this manner, various auxiliary devices may be particularly distributed in the controlled agricultural environment due to the flexible placement of communication ports on the imaging system.
In some implementations, the processor of the imaging system may be used to control one or more auxiliary devices and/or process data from the auxiliary devices. The processor may then utilize the data to adjust and control operation of one or more lighting fixtures (e.g., adjusting the PAR output from the lighting fixture) one or more coolant circuits (e.g., adjusting the fluid flow through the coolant circuit including the lighting loop, hydronics loop, and cooling loops), one or more fans, one or more dehumidifiers, or one or more air conditioners in the controlled agricultural environment. In some implementations, various environmental conditions are measured and controlled to provide target vapor pressure deficits in the environment.
It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and/or structurally similar elements).
Following below are a glossary of terms and more detailed descriptions of various concepts related to, and implementations of, methods, apparatus, and systems for multisensory imaging for controlled environment horticulture. It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in numerous ways. Examples of specific implementations and applications are provided primarily for illustrative purposes so as to enable those skilled in the art to practice the implementations and alternatives apparent to those skilled in the art.
The figures and example implementations described below are not meant to limit the scope of the present implementations to a single embodiment. Other implementations are possible by way of interchange of some or all of the described or illustrated elements. Moreover, where certain elements of the disclosed example implementations may be partially or fully implemented using known components, in some instances only those portions of such known components that are necessary for an understanding of the present implementations are described, and detailed descriptions of other portions of such known components are omitted so as not to obscure the present implementations.
In the discussion below, various examples of multisensory imaging systems, methods of acquiring, processing, and quantifying multisensory imagery, and inventive lighting systems integrated with multisensory imaging systems are provided, wherein a given example showcases one or more particular features in a given context. It should be appreciated that one or more features discussed in connection with a given example of a multisensory imaging system may be employed in other examples according to the present disclosure, such that the various features disclosed herein may be readily combined in a given system according to the present disclosure (provided that respective features are not mutually inconsistent).
Albedo: The term “albedo” refers to the ratio of radiosity from an object to the irradiance (flux per unit area) of an object. Thus, it is a measure of reflection of radiation out of the total radiation impinging on an object, measured on a scale from 0 (corresponding to a black body that absorbs all incident radiation) to 1 (corresponding to a body that reflects all incident radiation). The albedo of an object at a certain wavelength band or spectral region may be measured. For example, UV albedo refers to UV radiation reflected from an object out of the total UV radiation impinging on the object. As another example, narrowband albedo refers to narrowband radiation reflected from an object out of the total narrowband radiation impinging on the object.
Controlled Environment Horticulture: Controlled Environment Horticulture (CEH) (also referred to as controlled environment agriculture or CEA) is the process of growing plants in a controlled environment where various environmental parameters, such as lighting, temperature, humidity, nutrient levels, and carbon dioxide (CO2) concentrations are monitored and adjusted to improve the quality and yield of the plants. Compared to conventional approaches of plant cultivation, CEH may enable year-round production of plants, insensitivity to variable weather conditions, reduce pests and diseases, and reduce the number of resources consumed on a per plant basis. Additionally, CEH may support various types of growing systems including, but not limited to soil-based systems and hydroponics systems.
A controlled agricultural environment is typically enclosed, at least in part, by a building structure such as a greenhouse, a grow room, or a covered portion of a field in order to provide some degree of control over environmental conditions. One or more artificial lighting systems are often used in such controlled agricultural environments to supplement and/or replace natural sunlight that may be obstructed by the building structure or insufficient during certain periods of the year (e.g., winter months). The use of an artificial lighting system may also provide yet another measure of control where the intensity and spectral characteristics of the lighting system may be tailored to improve the photosynthetic rates of plants. Various types of artificial lighting systems may be used including, but not limited to, a high intensity discharge lamp, a light emitting diode (LED), and a fluorescent lamp.
Emissivity: “emissivity” is a measure of an object's ability to emit infrared energy. Emitted energy indicates the temperature of the object. Emissivity can have a value from 0 (shiny mirror) to 1 (blackbody).
Feature/Labeled Feature Set: a “feature” is a structured mathematical representation of a discrete measured value that is suitable for input into a machine learning system. Features determine what information a machine learning algorithm has access to regarding the measurement. A plurality of different discrete measurements may be used to develop a “labeled feature set” for a reference condition.
Field of View: a “field of view” refers to an area or footprint of inspection in which a measurement of some measurable property may be captured by one or more sensors/cameras.
Hyperspectral Imaging: “hyperspectral imaging” is an imaging technique that collects and processes a wide spectrum of radiation (either continuous or many discrete measurements) for each pixel in the plurality of pixels in the resulting image. Unlike multispectral imaging, which measures finite, non-continuous narrowbands, hyperspectral imaging measures continuous ranges of wavelengths over a wide spectrum.
Illuminator: an “illuminator” is a radiation source, such as an LED, that is not primarily used to provide radiation for sensing, in contrast to an irradiator. An illuminator may, for example, provide ambient lighting in an enclosed structure, or provide photosynthetically active radiation (PAR) in a CEH system.
Irradiator: an “irradiator” is a radiation source, such as an LED, that primarily provides light for sensing. For example, a series of narrowband irradiators may be used to irradiate an object to collect multispectral images of the object.
Multisensory Imaging: “multisensory imaging” is an imaging technique that collects and processes a plurality of imagery and sensory data to create a multisensory image, where each pixel in the plurality of pixels in the resulting image contains finite narrowband spectral data as well as sensory data. Each pixel in the resulting image may be close or far apart in any dimension.
Multispectral Imaging: “multispectral imaging” is an imaging technique that collects and processes a plurality of finite, non-continuous narrowband images, where each pixel in the plurality of pixels in the resulting image contains finite narrowband spectral data.
Narrowband/Narrowband Image: a “narrowband” is a narrow wavelength band of radiation with a bandwidth of about 2 nm and 40 nm. A narrowband image is an image captured using one or more narrowbands.
Normalize/Normalization: The terms “normalize”, or “normalization” refer to a process of modifying one or more disparate pieces of data relating to a same or similar thing, such that all of the pieces of data relating to the same or similar thing are homogenized in some manner (e.g., according to a predetermined standard or format).
Pixel: a “pixel” is the smallest element of an image that can be individually processed in an image. An image includes at least one pixel. Generally, each pixel in an image represents a radiation value at a spatial position in the field of view. Each pixel in an image may be close (neighboring) or far apart (with unmeasured space in between) in any dimension.
Radiation Value: a “radiation value” represents an amount of sensed radiation at a particular wavelength or spectral band. Each pixel in an image may be digitally represented by a radiation value. For example, a radiation value may be an amplitude of radiation at a particular narrowband reflected from an object within a camera's field of view. As another example, a radiation value may be an amplitude of fluorescence from an object within a camera's field of view.
Reference Condition/Reference Condition Library: a “reference condition” is a known condition for which a labeled feature set exists in a reference condition library. Examples of reference conditions for crops may include particular types of nutrient deficiency, insect infestation, fungal infection, or water stress. Other examples of reference conditions for crops may include stage of development, prospective crop yield, appearance, nutritional composition, structural integrity, flowering, and pollination. Reference conditions are not limited to crops and may describe known conditions of other objects in different environments. A reference condition library includes more than one labeled feature sets for various reference conditions. A machine learning algorithm may be used to compare experimental results to the reference condition library to determine if one or more reference conditions are present.
Overview of Multisensory Imaging Systems and Methods
In the depiction of
As discussed in greater detail below in connection with different implementation examples, the spatial arrangement of sensors 5080 may comprise a one-dimensional (1D), two-dimensional (2D), or three-dimensional (3D) array of sensor nodes. In one aspect, a given sensor node may be considered to be a picture element or “pixel” of an image of the field of view 5070, e.g., an element that is indexed (or “addressable”) in the 1D, 2D, or 3D array of sensor nodes. With the foregoing in mind, it should be appreciated that, generally speaking, the spatial arrangement of sensors thus may include a number of discrete sensors (or integrated sensor assemblies) positioned at respective indexed/addressable positions (pixels) in a 1D, 2D, or 3D spatial arrangement. Some examples of a 2D spatial arrangement of sensors that may constitute the spatial arrangement of sensors 5080 include a CCD, CMOS or microbolometer sensor array (the “imaging chip”) of a digital camera, as discussed in greater detail below.
With reference again to
As shown in
More specifically, as readily observed in
In the example of
To this end, in one implementation the image processor 5000 processes the plurality of mono-sensory images 5090 to generate a multisensory image, wherein respective pixels of the multisensory image may have multiple pixel values (measurement values) respectively representing the two or more measurable conditions that were sensed at a given spatial position. For example, considering for the moment a 2D array of sensor nodes and corresponding 2D array of pixels in an image of the field of view 5070, the notation below represents a first pixel P1 (x1, y1), in which a first measurable condition C1 that may be sensed in the field of view has a first measurement value M1 at the first pixel P1, and in which a second measurable condition C2 that may be sensed in the field of view has a measurement value M2 at the first pixel P1:
P1(x1,y1)=[M1,M2].
With the foregoing in mind, the reference condition library 5120 may include multiple labeled feature sets 5140 (a “collection” of labeled feature sets) respectively corresponding to various reference conditions of prospective interest in the field of view 5070 (some of which reference conditions may relate to an object or objects and others of which reference conditions may relate to states of the ambient) which depend, at least in part, on the two or more measurable conditions at respective spatial positions in the field of view. For example, the reference condition library 5120 may include a first labeled feature set 5140-1 representing a first reference condition and including a first feature set FS1 having a first label L1. Similarly, the reference condition library 5120 may include a second labeled feature set 5140-2 representing a second reference condition and including a second feature set FS2 having a second label L2, and may further include a third labeled feature set 5140-3 representing a third reference condition and including a third feature set FS3 having a third label L3, and so on (in
Each feature set FS1, FS2, FS3 . . . FSM in the collection 5140 of labeled feature sets represents a set of “features,” in which at least some respective features in the set correspond to respective measurable conditions that may be sensed by the spatial arrangement of sensors 5080. For each such feature, the feature set includes a reference value corresponding to one measurable condition of the respective measurable conditions. Thus, if as discussed above, a first measurable condition C1 may be sensed at a given spatial position in the field of view 5070 and a second measurable condition C2 also may be sensed at the given spatial position in the field of view 5070, respective feature sets FS1, FS2, FS3 . . . FSM in the collection 5140 of labeled feature sets include particular (and unique) combinations of reference values for the features C1 and C2 that, as a feature set, together represent the reference condition serving as the label for the feature set. Table 1 below provides some illustrative examples of a collection 5140 of labeled feature sets based at least in part on the measurable conditions C1 and C2, and corresponding example labels for reference conditions of prospective interest:
In one implementation, the image processor 5000 employs a variety of image processing and machine learning (ML) techniques to process a multisensory image to estimate or determine one or more environmental conditions of interest 5160 observed at respective spatial positions in the field of view 5070, based on the collection 5140 of labeled feature sets in the reference condition library 5120. In some implementations, various models are trained and utilized by the image processor 5000 to process multisensory images and “find” in those images one or more conditions that match, with a certain level of confidence, one or more corresponding reference conditions represented by the collection 5140 of labeled feature sets in the reference condition library 5120.
As would be appreciated by those of skill in the relevant arts, Machine learning (ML) is a branch of artificial intelligence based on the idea that systems (e.g., intelligent agents, like the image processor 5000) can learn from data, identify patterns, and make decisions with minimal human intervention. Thus, ML relates to algorithms and statistical models that intelligent agents (e.g., the image processor 5000) use to progressively improve their performance on a specific task. In more formal terms, an intelligent agent based on an ML model learns from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E. Deep learning is a subset of AI using layered structures of algorithms called an artificial neural network (ANN).
Machine learning tasks conventionally are classified into multiple categories. In “supervised learning,” an ML algorithm builds a mathematical model of a set of “training data” that contains both the inputs and the desired outputs from performing a certain task. For example, if the task were determining whether an image contained a certain object having a condition that corresponded to a reference condition of interest, the training data for a supervised learning algorithm may include a first image or images of the object having the particular condition of interest and a second image or images of the object not having the condition of interest (the input), and each image would have a label (the output) designating whether or not the object had the condition of interest. “Semi-supervised learning” algorithms develop mathematical models from incomplete training data, where a portion of the sample inputs are missing the desired output. “Classification” algorithms and “regression” algorithms are types of supervised learning. Classification algorithms are used when the outputs are restricted to a limited set of values (e.g., represented by the Boolean values one and zero), whereas regression algorithms have continuous outputs (e.g., any value within a range of values).
In “unsupervised learning,” an ML algorithm builds a mathematical model of a set of data which contains only inputs and no desired outputs. Unsupervised learning algorithms are used to find structure in the data, like grouping or clustering of data points. Unsupervised learning can discover patterns in the data, and can group the inputs into categories, as in “feature learning.” “Dimensionality reduction” is the process of reducing the number of “features” (e.g., inputs) in a set of data. “Active learning” algorithms access the desired outputs (training labels) for a limited set of inputs based on a budget and optimize the choice of inputs for which it will acquire training labels. When used interactively, these inputs can be presented to a human user for labeling (“annotation”).
Examples of various ML models known in the relevant arts include, but are not limited to, Linear Regression, Logistic Regression, Decision Tree, Support Vector Machine, Naive Bayes, kNN, K-Means, Random Forest, Convolution Neural Network, Multilayer Perceptron, and Recurrent Neural Network.
Accordingly, it may be appreciated from the foregoing that a multisensory imaging system may be implemented for a variety of applications and, in particular, CEH. As discussed in different examples below, a multisensory imaging system according to the concepts disclosed herein may be implemented to cover various sizes (and shapes) of fields of view 5070, and differing degrees of resolution for a spatial arrangement of sensors—in these respects, it should be appreciated that the general concepts underlying multisensory imaging systems disclosed herein are agnostic to the size/scale of field of view and resolution of images and, as such, may be implemented so as to cover different sizes/scales of fields of view with various resolution. With respect to CEH, multi sensory imaging systems according to the present disclosure may be employed to observe groups of crops, individual plants, or parts of plants, over various periods of time, to provide a wealth of information about the evolution of crops and their growing environment.
An Exemplary Multispectral Imaging System
In one example implementation, a multisensory imaging system according to the present disclosure is implemented more specifically as a multispectral imaging system that provides finite spectra sensing. In this manner, the imaging system may detect the presence and/or quantify reference conditions. These reference conditions may have a time-dependence and/or a spatial distribution within an environment. In another implementation, the imaging system may be integrated with a CEH system. In this manner, the multisensory imaging system may detect the presence and/or quantify certain reference conditions present in the CEH system.
The imaging system may be used to characterize reference conditions of objects within an environment. These reference conditions may, for example, be related to the growth and/or health of plants in an environment as a function of time. This may be accomplished, in part, by using the imaging system by itself or in conjunction with one or more light sources and/or illuminators to irradiate plants and/or other subjects of interest with different wavelengths of radiation and measure the spectral optical properties of the plants and/or other subjects of interest in their surroundings over time, in response to irradiation at different wavelengths. The foregoing process may be referred to as “kinetic finite absorbance and reflectance spectroscopy,” in which different finite narrowband spectral images and/or other information are collected for plants and/or other subjects of interest in response to irradiation at particular wavelengths, as a function of time, and then the acquired images/collected information are analyzed to determine reference conditions in the plants and/or other subjects of interest.
The flash controller 5020 (e.g., a microprocessor) may regulate the narrowband irradiators 1140 The flash controller 5020 may be configured so that each narrowband irradiator in the imaging system 5050A can be individually controlled. The flash controller 5020 may also be configured so that each narrowband irradiator can be activated for a brief period. In this manner, the flash controller 5020 may provide a means to sequentially flash or pulse each narrowband irradiator in the plurality of narrowband irradiators 1140.
The flash controller 5020 may be programmed to automatically activate one or more narrowband irradiators 1140 at set time intervals, or at certain times of day, for a set period of time. The flash controller 5020 may also be configured to receive communications from a remote device (e.g., a computer, a server, a tablet, or a smartphone) to determine narrowband irradiator control.
The flash controller 5020 or the image processor 5000A (e.g., a Raspberry Pi processor) may include one or more WiFi antennas and accompanying electric circuits (e.g., chipsets, processors) to facilitate wireless communication with the narrowband irradiators 1140 and other devices. In some implementations, the flash controller 5020 or the image processor 5000A may include a transmitter and/or a receiver to communicate with one or more lighting fixtures, environmental condition controllers, or remote devices (e.g., a computer, a server, a tablet, a smartphone, or lighting fixture controllers). In this manner, the flash controller 5020 or the image processor 5000A may control and/or coordinate with the lighting fixtures or other environmental condition controllers so that the imaging system 5050A can acquire images and/or collect information in a particular environmental setting. For example, the flash controller 5020 or the image processor 5000A may be configured to activate narrowband irradiators 1140 under dark conditions with the lighting fixtures turned off momentarily while measurements/images are acquired by the imaging system 5050A. As another example, the flash controller 5020 or the image processor 5000A may be configured to stop HVAC or fan operation during image acquisition so that objects are as still as possible.
In another example, the flash controller 5020 or the image processor 5000A may include a receiver to receive a signal from a remote device, which may include a command to adjust the operation of the narrowband irradiators 1140. Commands may include, but are not limited to, adjusting the time interval between images acquired and/or information sensed, the length of time of irradiation, the intensity of irradiation, the order of irradiation if more than one narrowband irradiator is to be flashed, and/or determining which irradiators are activated for a particular measurement/acquisition.
The flash controller 5020 may supply power to the irradiator. The flash controller 5020 may receive electrical power directly from an electrical power grid, or indirectly from another device. For example, the flash controller 5020 may receive electrical power from one or more lighting fixtures, or via a portable energy storage device, such as a rechargeable battery.
The flash controller 5020 may also regulate the one or more imagers/sensors 1005 so that the imagers/sensors acquire imagery and/or collect measurements at times relevant to irradiation. As an example, the flash controller 5020 may simultaneously activate a narrowband irradiator and an imager so that imagery is acquired while the narrowband irradiator is activated. In another example, the flash controller 5020 may activate the narrowband irradiator and the imager at different times. The imager may acquire imagery after the narrowband irradiator has been activated. In another implementation, the imagers/sensors 1005 may be controlled by another controller. In this case, the other controller may be coupled to the flash controller 5020 to coordinate the activation times of the irradiators 1140 and the imagers/sensors 1005.
The narrowband irradiators 1140, controlled by the flash controller 5020, provide radiation at wavelengths within a spectral range including UV, visible, NIR, SWIR, and LWIR regimes. Each narrowband irradiator may provide relatively narrowband radiation. For example, the spectral bandwidth may be about 2 nm to about 40 nm. In another implementation, each narrowband irradiator may provide substantially monochromatic radiation. The narrowband irradiators 1140 may be configured to turn on or off quickly and/or to respond quickly to signals from the flash controller 5020, so that each narrowband irradiator may provide a brief flash or pulse of radiation. Each brief flash may be up to 10 seconds in length.
In some implementations, the narrowband irradiators 1140 may be an array of LEDs elements. The spectral range of the LED array may include approximately 275 nm to approximately 2060 nm. In this manner, respective wavelengths of essentially monochromatic LED elements may include, but are not limited to, 275 nm, 365 nm, 440 nm, 450 nm, 475 nm, 500 nm, 530 nm, 620 nm, 630 nm, 660 nm, 696 nm, 730 nm, 760 nm, 850 nm, 860 nm, 940 nm, 950 nm, 1450 nm, 1610 nm, and 2060 nm. More than one LED element of a particular wavelength may be position around the imager/sensor 1005 to provide more uniform irradiation. For example, LED elements of a particular wavelength may be positioned on opposing sides of the imager/sensor 1005.
The irradiation profile for different narrowband irradiators may be substantially uniform. In some implementations, an optical element, such as a diffuser, or lens may be used to change the irradiation profile. In some implementations, each narrowband irradiator may provide radiation to substantially the same area. In this manner, the narrowband irradiators 1140 irradiating area may overlap with at least part of the overall field of view of the imagers/sensors 1005.
The narrowband irradiators 1140 may provide radiation to image and/or acquire information from one or more plants 900 or plant zones. In some implementations, it may be preferable for the narrowband irradiators 1140 to emit radiation with a sufficient intensity to acquire images/information at a desired quality (e.g., the signal-to-noise ratio of the image/information is above a pre-defined threshold) without causing chemical and/or morphological changes to the plant(s) (e.g., photomorphogenesis). In this manner, the various images/information acquired by the imager(s)/sensor(s) 1005 are representative of the plant(s) in their non-illuminated state. For example, LED irradiators 1140 may have a wattage rating less than about 6 Watts (the wattage rating may be correlated to the radiation output from the LED irradiators).
In some implementations, the narrowband irradiators 1140 may activate as a flash or brief pulse with a sufficient length to acquire images/information at a desired quality. This may be preferable in cases where it is desirable to reduce the time to acquire multiple images (e.g., ensuring the images are acquired at the same conditions). Short flashes may also prevent unwanted photochemical modifications to the plant, which may alter its optical properties.
The imaging system 5050A may generally include one or more imagers and/or one or more sensors 1005 to acquire imagery, video and/or spectra, respectively, of an object and/or an environment. In some implementations, the imagers/sensors 1005 may acquire imagery and/or sensory data from an object and/or an environment within a field of view 5070A of the imagers/sensors. In general, the imagers/sensors may acquire 2D imagery and/or 3D imagery (e.g., Lidar, a pan-tilt-zoom (PTZ) camera) of the environment. The camera/sensor may also have a field of view that ranges between a portion of the environment that includes one or more plants to an entire room of the environment. Note the environment may contain multiple rooms.
The imagers/sensors may acquire imagery and/or sensory data in response to irradiation of a plant zone 900 in a CEH system by the narrowband irradiators 1140. In some implementations, the imagers/sensors 1005 may acquire imagery and/or sensory data while one or more narrowband irradiators is activated. In some implementations, the imagers/sensors 1005 may acquire imagery and/or collect information immediately following irradiation by one or more narrowband irradiators 1140. In some implementations, the imagers/sensors 1005 may acquire imagery and/or sensory data from an object and/or an environment within a field of view 5070A without irradiation of the object and/or environment by the narrowband irradiators 1140.
The imager(s)/sensor(s) 1005 may generally acquire imagery/spectra at wavelengths in the UV, visible, NIR, SWIR, and LWIR regimes. For example, the imagers/sensors 1005 may include cameras that acquire imagery in various spectral bands including, but not limited to the ultraviolet band (e.g., wavelengths between 10 nm and 400 nm), the visible band (e.g., wavelengths between 400 nm and 700 nm), the near infrared (NIR) band (e.g., wavelengths between 700 nm and 1.4 μm), the mid infrared (MIR) band (e.g., wavelengths between 1.4 μm and 8 μm), and the far infrared (FIR) band (e.g., wavelengths greater than 8 μm). The imagers/sensors may preferably acquire imagery/spectra at wavelengths between approximately 275 nm to approximately 2060 nm and between approximately 8 μm to approximately 14 μm. In some implementations, the imaging system 5050A may include a first camera to acquire imagery from the UV to short wavelength infrared (SWIR) regimes and a second camera to acquire thermal imagery using the long wavelength infrared (LWIR) regime. In some implementations, the second camera may continually acquire LWIR thermal imagery while the first camera is periodically activated in combination with the narrowband irradiators 1140 to acquire UV-SWIR imagery.
The imaging system may acquire imagery/spectra under various lighting conditions. As described above, imagery may also be acquired while other radiation sources in the environment are deactivated. For example, light sources in a lighting fixture may be turned off while acquiring imagery/spectra with the imaging system. In some implementations, imagery may be acquired while other radiation sources are active. In some implementations, a background subtraction may be applied to remove the radiation emitted by the other radiation sources. For example, imagery/spectra may be acquired with and without radiation from the LED element in the LED array of the imaging system. If the radiation does not cause substantial changes to the plant's optical properties (e.g., alters the photochemistry of the plant), the portion of light reflected by the radiation from the LED element may be extracted by taking a difference between the two images/spectra. In another example, the LED source may have a known radiation spectrum based, in part, on the number and type of LED elements used. The spectra may be calibrated according to various standards and/or guidelines in plant research (e.g., the International Committee for Controlled Environment Guidelines). If the location and orientation of the LED source-relative to the plants and the imaging system are known, the radiation from the LED source may be subtracted directly from the imagery/spectra acquired when the LED element of the LED array and the LED source are both active.
In some implementations, one or more of the imagers may be cameras with sufficiently high pixel resolution (e.g., an 8-megapixel camera, imagery is acquired at 4K resolution). The imager(s)/sensor(s) may also have a spectral resolution between about 2 nm to about 40 nm. In other implementations, imagers/sensors with different spectral resolutions may be used.
The imager(s)/sensor(s) 1005 may acquire the spectral information from an object and/or an environment within the field of view 5070A at different wavelengths by adjusting the wavelength of light irradiating plants 900. For example, the narrowband irradiators 1140 may illuminate the plants 900 with substantially monochromatic radiation 5040, the imagers/sensors 1005 may acquire images and/or spectra corresponding to reflected light 5060 reflected from the plants 900, thus the imagery/spectra 5090A acquired may correspond to the particular wavelength of the radiation. This process may be repeated sequentially with radiation with several narrowband irradiators 1140. In this manner, the imaging system 5050A may acquire a series of narrowband images, each narrowband image corresponding to a particular narrowband irradiation 5040. Here, the imagers/sensors 1005 may acquire multisensory imagery without the use of spectral filters.
The imagery and/or sensory data collected by the imaging system 5050A may include at least one pixel, where a pixel is the smallest element that can be individually processed. A narrowband image may include a plurality of pixels, where each pixel in the narrowband image represents a radiation value at a spatial position in the field of view 5070. Data collected from a point sensor may represent a single pixel.
The imaging system 5050A may include one or more image processors 5000A. The image processor 5000A may be coupled to the imagers/sensors 1005 to receive imagery and sensory data 5090A acquired by the imagers/sensors 1005. In some implementations, the image processor 5000A may process several narrowband images 5090A to produce a multispectral image. In some implementations, the image processor 5000A may process imagery and/or sensory data to observe one or more conditions 5160A at respective spatial positions in the field of view 5070A. A reference condition library 5120A and/or an algorithm may be used to observe conditions 5160A in the imagery and/or sensory data. Image processing may extract morphological data from the acquired imagery and/or sensory data and integrate it for a wide range of applications. These concepts are described in greater detail below.
In some implementations, the camera(s)/sensor(s) 1005 may be configured to acquire sensory data proximate to the portion of the plants and/or other subjects of interest in the environment in the CEH system irradiated by the narrowband irradiators 1140. In some example implementations employing multiple imagers/sensors 1005, the multiple imagers/sensors 1005 may be co-located (e.g., in sufficient proximity to one another) such that the respective fields of view (FOV) of the cameras and/or sensors are substantially overlapping or substantially the same. In this manner, different types of sensory data may correspond to the same region of the environment, thus enabling a more comprehensive analysis of the environment. In some implementations, the portion of the plants and/or other subjects of interest irradiated by the narrowband irradiators 1140 may be further subdivided into subregions that are each characterized by corresponding sets of cameras/sensors 1005 disposed on/integrated in the imaging engine 1100.
The imaging system 5050A may be designed to be operated remotely or automatically. Narrowband irradiator elements 1142 of the same wavelength may switch on at the same time automatically at certain preprogrammed time intervals. Alternatively, narrowband irradiator elements 1142 of the same wavelength may be activated remotely. The flash controller 5020, either alone or in communication with a remote device, may control automatic and/or remote narrowband irradiator activation. An image may be acquired (and/or information may be collected) automatically using imager(s)/sensor(s) 1005 each time narrowband irradiators are activated. Supplemental illumination in the CEH may be turned off automatically when images are acquired. The imaging system may acquire an image at each available narrowband or combinations of narrowbands at each measurement interval. Alternatively, the imaging system may acquire a subset of images at only some of the available narrowbands or combinations of narrowbands at each measurement interval.
In some implementations, the LED elements 1142 respectively may be activated for a relatively short time period (i.e., turning on and off quickly) in succession (and optionally according to some pattern or order), thus exposing the plants to a brief “flash” of light when acquiring various information relating to reflected radiation using the camera(s)/sensor(s) 1005. For example, the LED elements 1142 may emit radiation for a duration of less than about 1 second. Activating the LED elements 1142 in this manner may have multiple benefits including, but not limited to (1) reducing the time delay between acquiring images/information at different wavelengths so that the multiple images/information acquired are representative of the same environmental conditions and (2) reducing the duration in which the plants and/or other imaging subjects are exposed to radiation. In some implementations, the camera(s)/sensor(s) 1005 may be synchronized with the LED elements 1142 such that the camera(s)/sensor(s) 1005 is/are triggered to acquire an image/information when the LED elements 1142 are activated. In this manner, a series of images/information may be collected by sequentially flashing the plants with radiation from different LED elements 1142 and capturing an image/information during each flash using the camera(s)/sensor(s) 1005. In yet other implementations, multiple LEDs having different spectral outputs may be activated together while one or more images and/or other information is acquired relating to radiation absorbed and/or reflected by the irradiated plants and/or other subjects.
The imaging engine 1100 may include one or more cameras, other imaging devices (e.g., a thermal imager), or other sensors (collectively referred to with reference number 1005) disposed in or on (integrated with) the imaging engine 1100. The imager(s)/sensor(s) 1005 may be used to acquire various information about the agricultural environment including, but not limited to imagery (video imagery or still imagery, as well as thermal imagery) of the plants and/or other subjects of interest in the environment in the CEH. Examples of various types of sensors that may be included in the imager(s)/sensor(s) 1005 include, but are not limited to, one or more cameras responsive to radiation in a range of at least visible wavelengths and/or IR wavelengths, an air temperature sensor, a near infrared (NIR) leaf moisture sensor, a relative humidity sensor, a hyperspectral camera, a carbon dioxide sensor, an infrared (IR) leaf temperature sensor, an airflow sensor, and a root zone temperature sensor.
In some implementations, the imaging engine 1100 in
One example of the camera/sensor 1005A includes, but is not limited to, the Raspberry Pi Camera Module v2. The v2 Camera Module has a Sony IMX2198-megapixel sensor and may be used to acquire high-definition video and/or still photographs. The sensor supports 1080p30, 720p60, and VGA90 video modes in addition to still capture. The sensor attaches to the camera serial interface (CSI) port on the Raspberry Pi via a 15 cm ribbon cable. The camera works with various Raspberry Pi models including, but not limited to the Raspberry Pi 1, 2, and 3. The camera 1005A may be accessed and controlled using the multimedia abstraction layer (MMAL) and video for Linux (V4L) API's. Additionally, numerous third-party software libraries may be used to control the camera 1005A in various software environments (e.g., Python using the Picamera Python library).
Another example of the camera/sensor 1005A includes, but is not limited to, the infrared Camera Module v2 (Pi NoIR). The v2 Pi NoIR has a Sony IMX219 8-megapixel sensor, which is the same as the camera used in the Raspberry Pi Camera Module v2. The difference is that the Pi NOIR does not include an infrared filter (NOIR=No Infrared) and is thus able to acquire imagery of at least a portion of the infrared spectrum (e.g., NIR). In some implementations, the Pi NoIR may be used together with a square of blue gel to monitor the health of green plants. Similar to the Pi Cam, the Pi NoIR may with various Raspberry Pi models including, but not limited to the Raspberry Pi 1, 2, and 3. Also, the Pi NoIR camera may also be accessed and controlled in software using the MMAL and V4L API's as well as third-party libraries (e.g., Python using the Picamera Python library).
The camera/sensor 1005B may be a longwave IR thermal imager responsive to wavelengths in a range of from approximately 8 micrometers to approximately 14 micrometers (LWIR). One example of such a thermal imager includes, but is not limited to, the FLIR Lepton 3.5 micro thermal imager, which provides 160×120 pixels of calibrated radiometric output.
One example of the IR single point sensor 1005C includes, but is not limited to, the Melexis MLX90614 infrared thermometer for non-contact temperature measurements. An IR sensitive thermopile detector chip and the signal conditioning application-specific integrated circuit (ASIC) are integrated in the same TO-39 can. The MLX90614 also includes a low noise amplifier, 17-bit analog-digital converter (ADC), and a powerful digital signal processor (DSP) unit to achieve a high accuracy and resolution for the thermometer. The thermometer may be factory calibrated with a digital SMBus output providing access to the measured temperature in the complete temperature range(s) with a resolution of 0.02° C. The digital output may be configured to use pulse width modulation (PWM). As a standard, the 10-bit PWM is configured to continuously transmit the measured temperature in range of −20° C. to 120° C., with an output resolution of 0.14° C.
One example of the proximity sensor 1005D includes, but is not limited to, the VL53L1X time-of-flight proximity sensor. This sensor is a single point laser rangefinder with a field of view of 27°. The proximity sensor has a 940 nm emitter and a single photon avalanche diode. It has a programmable region-of-interest (ROI) size on the receiving array, allowing the sensor field of view to be reduced. The programmable ROI position on the receiving array provides multizone operation.
One example of the quantum sensor 4220E includes, but is not limited to, an Apogee SQ-520 or LI-190R. The quantum sensor is used to measure photosynthetically active radiation (PAR) at various points in the CEH, including, but not limited to above the grow canopy, at the grow canopy and reflected PAR. The Apogee SQ-520 is a full spectrum PAR sensor. The SQ-500 series quantum sensors consist of a cast acrylic diffuser (filter), interference filter, photodiode, and signal processing circuitry mounted in an anodized aluminum housing, and a cable to connect the sensor to a measurement device. SQ-500 series quantum sensors are designed for continuous PPFD measurement in indoor or outdoor environments.
In one exemplary method, the imagery/spectra at different wavelengths may be acquired in a serial manner by utilizing the following steps: (1) activating an LED element in the LED array of the imaging system to irradiate the plants with radiation at a first wavelength, (2) acquiring imagery/spectra using the camera(s)/sensor(s) in the imaging system, and (3) deactivating the LED element. These steps may be performed in sequence to acquire imagery/spectra at different wavelengths. For example, the imaging system may acquire images for the following cases: (1) illumination under 275 nm and 365 nm radiation, (2) illumination under visible radiation corresponding to various photosynthetic and terpene compound peaks, (3) illumination under NIR at 940 nm and 1450 nm to assess water content, (4) illumination under SWIR for detection of gases and other compounds, and (5) a LWIR heat map, which may be a acquired without activation of a light source.
The LED array 1140 may include one or more LED elements 1142. Each LED element 1142 of the array 1140 may emit radiation at a particular band of wavelengths or an essentially monochromatic wavelength and may be controlled independently from the other LED elements 1142. When one or more LED elements 1142 are operated to irradiate a desired portion of the environment (e.g., the plants below the lighting fixture 1000) with relatively narrow band or substantially monochromatic radiation, one or more of the cameras/sensors 1005 (e.g., camera 1005A) acquires a corresponding image that contains radiation reflected or otherwise emitted by the plant subjects in the field of view in response to exposure to radiation at the corresponding wavelength(s) of the operated LED element(s). Different LED elements 1142 may be activated to illuminate the desired portion of the environment with radiation at different wavelengths and the cameras/sensors 1005, in turn, may acquire corresponding images or other sensed information relating to reflected and/or emitted radiation resulting from the respective different wavelengths/wavelength bands of the activated LED elements. In some example implementations, after acquiring images and/or other information at multiple wavelengths/wavelength bands, a multispectral image may be formed by aligning and superimposing the respective acquired images onto each another. In this manner, the multispectral image may include spatial and spectral information regarding the desired portion of the environment (e.g., each pixel of the multispectral image contains corresponding spectra).
The imaging engine 1100 may also include supplementary LED arrays 1150A and 1150B (collectively referred to as supplementary LED array 1150) to augment the LED array 1140 and/or to alter the chemical/morphological properties of the plants. The imaging engine 1100 may also include power electronic circuitry 1160 to support the operation of the LED arrays 1140 and 1150 and cameras/sensors 1005. The imaging engine 1100 may also include a flash controlled 5020 (e.g., a Raspberry Pi processor) to control the LED arrays 1140 and 1150 and/or the cameras/sensors 1005. The imaging engine may also include one or more Power over Ethernet (PoE) ports 1184 and/or one or more Universal Serial Bus (USB) ports as communication and/or power ports.
The imaging engine 1100 may generally include one or more LED arrays 1140. Each LED array 1140 may include one or more LED elements 1142. For instance, each LED array 1140 may include between about 1 to about 100 LED elements 1142. The LED elements 1142 in the LED array 1140 may be disposed proximate to each other on the circuit board 1110. The LED arrays 1140 may be arranged on the circuit board 1110 to provide a desired illumination profile. For example, the LED arrays 1140A and 1140B may include the same type of LED elements 1142, thus providing multiple radiation sources that emit radiation at the same wavelength.
The LED array 1140 may generally include LED elements 1142 that respectively emit radiation at different wavelengths. For example, the LED elements 1142 may emit radiation at wavelengths ranging between about 200 nm to about 2 μm. The number of LED elements 1142 and the wavelengths at which they emit light may be chosen, in part, based on known spectral absorption and/or reflection peaks of various chemical compounds associated with the plants (see
The respective wavelengths of the radiation emitted by the LED elements 1142 may cover UV, visible, NIR, and SWIR regimes. In one example implementation, respective wavelengths of essentially monochromatic LED elements 1142 of the LED array 1140 may include, but are not limited to, 275 nm, 365 nm, 440 nm, 450 nm, 475 nm, 500 nm, 530 nm, 620 nm, 630 nm, 660 nm, 696 nm, 730 nm, 760 nm, 850 nm, 860 nm, 940 nm, 950 nm, 1450 nm, 1610 nm, and 2060 nm. More generally, the LED elements 1142 of the LED array 1140 may have radiation wavelengths between approximately 275 nm to approximately 2060 nm.
The supplementary LED array 1150 may include additional LED elements 1152. The LED elements 1152 may have one or more of the same features as the LED elements 1142 described above. In one example, the LED elements 1152 may emit radiation at one or more of the same wavelengths as the LED elements 1142 in order to increase the overall intensity of radiation when acquiring images/information relating to the irradiated plants/other subjects (i.e., both LED elements 1142 and 1152 are activated). In some implementations, the LED elements 1152 may provide a radiation output greater than the LED elements 1142. For example, the LED elements 1152 may have a wattage rating greater than about 6 Watts. The higher radiation output provided by the LED elements 1152 may be used, in part, to intentionally induce chemical and/or morphological changes to plants in the environment. For example, the LED elements 1152 may provide a higher radiation output at 730 nm in order to alter the day/night cycle of the plants (e.g., changing when the plant blooms). In another example, the LED elements 1152 may provide UV light to ward off pests in the environment.
The housing 1120 may be used, in part, to enclose and protect the various components of the imaging engine 1100 and to facilitate installation of the imaging engine 1100 onto the frame a lighting fixture in the CEH. In some implementations, the housing 1120 may form a substantially sealed enclosure in order to prevent moisture and/or water from contacting the various electronics, cameras, and sensors on the circuit board 1110. The housing 1120 may include a groove along its periphery to support a gasket 1124 and through-mounting holes 1122 for ease of installation. When the housing 1120 is installed in the CEH system, the gasket 1124 may deform to form a seal. In some implementations, the housing 1120 may form a substantially watertight seal.
The housing 1120 may be formed from various plastic and/or ceramic materials. In some implementations, the housing 1120 may be formed from a material that is substantially transparent to light at wavelengths corresponding to at least the emission wavelengths of the LED elements 1142 and 1152. Thus, radiation emitted by the LED elements 1142 and 1152 may transmit through the housing 1120 when irradiating the plants and/or the surrounding environment. In some implementations, the housing 1120 may be shaped to redirect radiation emitted by the LED elements 1142 and 1152 along a desired direction. For example, the housing 1120 may be shaped to redirect radiation emitted at wider angles towards the plants disposed directly below the imaging engine 1100 in order to more efficiently use the radiation for imaging/information acquisition. In some implementations, the surface finish of the housing 1120 may be altered to disperse radiation (e.g., a substantially smooth finish to provide specular illumination or a substantially rough finish to provide diffuse illumination).
In some implementations, the housing 1120 may be formed from a material that is not sufficiently transparent across the wavelength range of interest. For example, the camera 1005A may acquire imagery/information from the UV to SWIR ranges while the camera 1005B may acquire imagery/information in the LWIR range. Materials are typically not transparent across such a large wavelength range. Furthermore, in some instances parasitic absorption by the housing 1120 may affect the data collected by the camera(s)/sensor(s) 1005. In view of the foregoing, the housing 1120 may include multiple openings 1126 disposed near the camera(s)/sensor(s) 1005 that are shaped to support various optical elements tailored for the appropriate wavelength ranges of each camera/sensor 1005.
For example,
The image processor 5000A may perform one or more image processing steps. In some implementations, the imaging system 5050A may use one or more calibration references to facilitate calibration by the image processor 5000A of imagery and/or sensor data acquired by camera(s)/sensor(s) 1005. For example, a phantom may be used to calibrate imagery/spectra in the UV, visible, NIR, and SWIR regimes. The phantom may be an object with known optical properties including, but not limited to a known emissivity, absorptivity, and reflectivity at various wavelengths. In general, the phantom may have optical properties that vary as a function of wavelength or optical properties that remain substantially unchanged at different wavelengths. The phantom may have various shapes including, but not limited to a sphere, a polyhedron, a plant, a fungus, and an animal (e.g., a mammal, a fish). The phantom may also be dimensioned to have an overall size that is smaller, comparable, or larger than the plants being imaged.
By placing a phantom near the plants being imaged, the phantom can calibrate both the magnitude and wavelength of the imagery/spectra being measured. In some implementations, multiple phantoms may be deployed within the field of view of the camera(s)/sensor(s) to provide multiple references. For such cases, the multiple phantoms may be used to correct for image distortion (e.g., spherical aberration of the image) and/or the angle at which radiation is received by the camera(s)/sensor(s) (e.g., the camera/sensor may have a responsivity that varies as a function of the angle of incidence of the detected radiation).
As another example, thermal images acquired by the LWIR thermal camera(s) may be corrected using the crop water stress index (CWSI). CWSI may be extracted from a thermal image and may assist in compensating variability in the environmental parameters. Air temperature measurements using a more precise temperature sensor are used to calculate CWSI. CWSI is defined as:
where Tc is the average temperature in the region of interest, Ta is the air temperature measured by the more precise temperature sensor, Tc,min is the lowest pixel temperature within the region of interest in the thermal image, and Tc,max is the highest pixel temperature within the region of interest in the thermal image. The resulting CWSI values may be used to sense transpiration and/or stomatal conductance.
The size of pixels in a multispectral image may be defined by the highest resolution camera. Images and/or sensory data acquired by an imaging engine may have a substantially overlapping field of view so that they may be easily overlaid.
Generally, a multispectral image generated by the imaging system 5050A includes a plurality of pixels. Each pixel has a vector of values. The vector of values for a particular pixel may represent the respective reflectance at each narrowband irradiator wavelength. The vector of values for the particular pixel may also represent a temperature corresponding to the LWIR heat map.
The vector of values may provide the basis for a feature set for a machine learning or deep learning algorithm to facilitate identification of conditions of interest 5160A. The values measured at each pixel may be featurized so that they are suitable for input into an algorithm. In other words, the measured pixel values may be represented by a set of structured mathematical representations called features. In this way an algorithm may identify conditions of interest 5160A from real-time measurements, non-destructively and in situ, on a pixel-by-pixel basis. Particular values in a given feature set representing a condition of interest can be labeled (“annotated”) and thereby facilitate model training for use in machine learning matching (e.g., with a confidence value) to identify conditions of interest observed in real time measurements of an object (e.g., plant(s)). In the example with four narrowband spectral images and a thermal heat map, each pixel in the resulting multisensory image may correspond to a feature set that includes measured values for four narrowband spectral images and a thermal heat map.
where Ri is a reflectance value from a pixel at a particular wavelength, and R1, R2, R3, . . . Rn are the reflectance values from the pixel at each wavelength in the multisensory image. The image processor 5000A may convert the color spaces of imagery and/or sensory data. Each color space defines color with different attributes that can provide information for different applications. Different color spaces present this information in ways that are more convenient to make some calculations, or images may appear brighter and easier to distinguish. For example, imagery acquired using a RGB color space may be converted to another color space, such as HSV, HIS, or LAB. The image processor 5000A may process images acquired by the imaging system 5050A to remove background, shadow, and/or exterior objects in the images using thresholding, machine learning, and/or image processing algorithms. This process includes linear and non-linear image filtering for smoothing, sharpening, measuring texture, denoising, image compression, and image registration. Morphological operations are performed to change the foreground region via union operations. This process cleans up the results from thresholding. Structuring elements such as masks may also be used for morphology determination. Blobs or connected components may be extracted.
In some implementations, the image processor 5000A may determine vegetative indices (VIs) using the narrowband images 5090A. In this way, each pixel in a multispectral image may have one or more feature sets with one or more VI value. For example, the image processor 5000A may generate normalized difference spectral indices (NDSIs) that represent every possible coupled combination of narrowband reflectance wavelengths according to:
where R is the measured reflectance, and i and j refer to specific spectral bands. NDSIs may be novel combinations of spectral bands generated by spectral ratios. Other VIs may be generated, including NDVI, GNDVI, RNDVI, NNIR, MCARI, and RENDVI.
The reference condition library 5120A can include labeled feature sets corresponding to known reflectance or albedo values of objects of interest. The library 5120A may include respective spectra of chemical constituents of interest. The values of respective features in the labeled feature set may be discreet reflectance (or absorbance) values lifted from the known spectra at the particular narrowband wavelengths serving as the features in the feature set (e.g., the wavelengths of the narrowband irradiators 1140). The known spectra may be albedos (reflectance) or absorbance spectra. Known absorbance spectra may be converted to reflectance spectra using conversions known in the art (e.g., Reflectance=(1/Absorbance)). Values from known spectra may also be normalized before use as a labeled feature set.
The method of extracting discrete values from known spectra to create labelled feature sets may be applied to any known spectra of interest. For example,
The image processor 5000A may detect various conditions of interest using the acquired imagery and/or sensory data and the reference condition library. Spectral and sensory features measured by the imaging system 5050A may represent certain plant properties. These measured features may be used to characterize the plants by comparing them to labeled feature sets. The measured features may be correlated with reference conditions in a reference condition library 5120A. For example, the image processor 5000A may identify conditions such as plant type, growth stage, disease, nutrient deficiencies, nutrient related traits, chemical composition, chlorophyll estimation, and biotic and abiotic stresses. Certain parts of plants (e.g., flowering sites) and/or morphological characteristics may also be identified in an image.
As another example, the image processor 5000A may detect various chemical compounds based on their respective peak absorptance and/or peak reflectance including, but not limited to, mold, mildew, photosynthetic compounds, water, NO3, NO2, P4, K+, C2H4, CH4, O2, CO2, and thermal radiation (e.g., LWIR radiation). The presence of these compounds may vary between different plant species. For a particular plant species, the amount of these compounds relative to a nominal baseline, as measured by the imaging system 5050A, may provide valuable information on various aspects of the plant's development including, but not limited to, the stage of development, prospective crop yield, appearance, nutritional composition, structural integrity, flowering, and pollination.
The data collected by the imaging system 5050A may be used to monitor the development of plants and/or to provide feedback to adjust other components of the CEH (e.g., the total intensity or spectral intensity of the light emitted by the illuminating light sources) in order to improve the health and growth of the plants. For example, if the imaging system 5050A detects damage to the plants caused by pests, the illuminators may be adjusted to illuminate the plants with more UV light as a form of repellant. In another example, the imaging system 5050A may detect the presence of mildew in the environment. In response, illuminators may increase the amount of UV light at 275 nm as a countermeasure. In another example, the imaging system 5050A may acquire data over time to assess changes to the plant during a typical day/night cycle (e.g., blooming for short day/long day plants). This information may be used to alter when the plant blooms by adjusting the illuminators to illuminate the plants with more/less near infrared light (e.g., 730 nm light). In this manner, plants may be grown at a faster rate. The imaging system 5050A may also characterize the morphology of the plants, which in turn may be modified by illuminating the plants with radiation at different wavelengths (e.g., 730 nm wavelength radiation).
The processor 5000A may also be used to manage data communications (e.g., wired communication via Ethernet cables or wireless communication, including sending control signals to the imagers/sensors 1005 and receiving imagery and sensory data measured by the imagers/sensors 1005 for processing and/or transmission to a remote device (e.g., a remote computer or server). Acquired images and/or sensory data may be stored locally or on a remote server.
The imaging system 5050A may include more than one imaging engine 1100 to image/sense a greater portion of the environment. For example,
In some implementations, the various camera(s)/sensor(s) 1005 may acquire imagery 5090 at different fields of view 5070. Thus, the images may be orthographically corrected prior to stitching to ensure the multisensory image does include distortions cause by the different fields of view. Said in another way, the images may be orthographically corrected to ensure each of the images represents a substantially similar orthographic view of the imaged object(s) and/or environment.
In some implementations, the one or more imaging engines 1100 may be stand-alone modules or may be integrated with other components of the CEH system.
One or more imaging engines may be integrated with or disposed on lighting fixtures 1000 providing artificial lighting in a CEH system 2000. In this case, the lighting fixtures 1000 may define a (x,y) coordinate plane that may be used to index the position of each imaging engine and the acquired imagery. Alternatively, an imaging engine may be coupled to a lighting fixture 1000 via a wired connection. For example, an imaging engine may be coupled to the ports of a lighting fixture 1000 via a PoE cable or USB cable. In another example, the imaging engine may be coupled to a remote device (e.g., a computer, a server) that controls one or more illuminators via a separate wired connection. The imaging engine may also be coupled to an illuminator or a remote device via a wireless connection.
In one aspect, a stand-alone imaging engine may provide greater ease in installation and replacement. For example, the imaging engine may be readily connected (or disconnected) from a lighting fixture 1000. The plug-and-play configuration allows the imaging engine to be installed at any desired lighting fixture 1000 in the CEH system 2000. In some implementations, the imaging engine may also be disconnected from a lighting fixture for maintenance or replacement with an updated and/or different imaging engine.
In another aspect, the imaging engine may provide greater flexibility in terms of deployment in an environment. For example, the imaging engine may be installed for a subset of the lighting fixtures 1000 present in the environment depending on the desired coverage of the environment. The imaging engine may also be installed without being constrained to the locations of lighting fixtures in the CEH system. Thus, the coverage provided by the lighting fixture and the imaging engine may be decoupled. Additionally, the imaging engine may be oriented such that the field of view covers the environment from different perspectives (e.g., a side view of the plant stems, a top-down view of the plant leaves). Different perspectives may provide a means for acquiring 3D imagery.
An Exemplary Multisensory Imaging System Using a Distributed Sensor Grid
In some implementations, a multisensory imaging system may provide an array of sensors integrated into the CEH system to collect various types of measurements. The sensors may be arranged spatially within the CEH system to collect measurements from a large portion of the CEH system.
In one exemplary implementation, multiple sensors are distributed in a CEH system as a distributed sensor grid. The distributed sensor grid includes one or more node arrays, where each node array divides at least a portion of the controlled agricultural environment into nodes, e.g., discrete points in space which have a known location (e.g., absolute or relative) in the environment. In various aspects, a given node array of a distributed sensor grid may be one dimensional, two dimensional, or three dimensional (e.g., based at least in part on the distribution of growing areas and/or crops in the controlled agricultural environment). For example, in some implementations, a given node array may include multiple nodes arranged in a substantially linear or curvilinear fashion spaced along a row of plants to provide a one-dimensional node array. Another type of node array may include multiple nodes arranged in a horizontal plane substantially parallel to a floor or a ceiling in the CEH system to provide a two-dimensional node array. Yet another type of node array may include multiple nodes arranged in multiple horizontal planes substantially parallel to the floor or ceiling in the CEH system, wherein the respective horizontal planes of nodes constitute multiple vertical levels corresponding to different zones of interest in the controlled growing environment (e.g., the soil, the plant, the lighting canopy, and the ambient environment).
The distributed sensor grid may act as a compound-eye in the CEH system. Each sensor node may act as a spatially-indexed pixel, and together the pixels may form a compound-eye multisensory image. The dimensionality of the node array determines the dimensionality of the multisensory image. For example, a node array arranged in a substantially linear fashion along a row of plants provides a 1D multisensory image. A node array arranged in a horizontal plane substantially parallel to a floor in the CEH system provides a 2D multisensory image. A node array arranged in multiple horizontal planes substantially parallel to the floor in the CEH system provides a 3D multisensory image.
Node Array
Each node array 4100 covers at least a portion of an agricultural environment. In some controlled agricultural environments, one node array may be sufficient given a particular number and arrangement of plants in a growing area, while in other environments multiple node arrays may be employed to flexibly configure a distributed sensor grid (in some instances over multiple growing areas in the environment with different layouts and/or different crops). For example, in vertical farming (in which different growing areas are stacked one on top of another in a vertical arrangement), one or more node arrays 4100 can be used for each vertically-stacked growing area in the environment. In another example, an agricultural environment can be divided into separate climate-controlled rooms with each room having one or more node arrays 4100. Each node array 4100 divides the covered portion of the agricultural environment into a grid of nodes 4200, where each node 4200 is a discrete point with a known coordinate location within the node array 4100. As noted above, respective nodes 4200 can include one or more sensors 4220 to monitor growth conditions proximate to a given node (e.g., in a volume of space around the node, which may depend in part on the type(s) of sensor(s) deployed at the node). In some implementations, the number of nodes 4200 in a node array 4100 can depend upon the constraints imposed on or by the agricultural environment.
The coordinate location of each node 4200 can include one or more coordinate components to describe the location of a node 4200 in the agricultural environment. In some implementations, the coordinate location of a node 4200 can correspond to a physical location in the agricultural environment with reference to a spatial origin. For example, the corner of a growing area can be set as the origin of the coordinate system and nodes 4200 can be defined at known and/or defined distances from the origin along one or more axes (e.g., respective X, Y, and Z axes). In some implementations, the coordinate location can correspond to an indexed location related to one or more aspects of the physical arrangement of the agricultural environment (e.g., dimensions and/or shape of one or more growing areas, arrangement of plants in a given growing area, arrangement of control systems in a given growing area).
Sensors
One or more sensors 4220 can be deployed at a particular node 4200 to monitor parameters relevant to growth conditions. The sensors 4220 can include, but are not limited to, a visible light sensor, a UV light sensor, an air temperature sensor, a relative humidity sensor, an airflow sensor, a CO2 sensor, an IR temperature sensor, a chemical sensor, a pH sensor, and cameras configured to capture still images or videos of the agricultural environment with various spectral qualities, as described above. In some implementations, multiple sensors 4220 can be packaged into an integrated sensor assembly to simplify wiring and ease of installation. Each node 4200 in a node array 4100 can also include different combinations of sensors 4220 pertinent to the region of the environment the node 4200 is located in. For example, different types of sensors 4220 may be deployed according to the object of interest in an environment.
Multisensory Imagery
Generally, the multisensory image generated by the imaging system 5050B includes a plurality of pixels (nodes). Each pixel has a vector of measured values corresponding to each sensor type. The vector of values may provide the basis for a feature set for a machine learning or deep learning algorithm to facilitate identification of conditions of interest 5160B. The values measured at each pixel may be featurized so that they are suitable for input into an algorithm. In other words, the measured pixel values may be represented by a set of structured mathematical representations called features. In this way an algorithm may identify conditions of interest 5160B from real-time measurements, non-destructively and in situ, on a pixel-by-pixel basis. Particular values in a given feature set representing a condition of interest can be labeled (“annotated”) and thereby facilitate model training for use in machine learning matching (e.g., with a confidence value) to identify conditions of interest observed in real time measurements of an object (e.g., plant(s)).
In the example illustrated in
The nodes 4200 in the node array 4100 can also be configured to share power and network connections to simplify the integration of multiple sensors 4220 in the distributed sensor grid 4000. In some implementations a plurality of lighting fixtures 1000 can be used as a connectivity platform for the distributed sensor grid 4000. Sensors 4220 can couple to PoE ports or the USB ports for power and networking using cables or dongles. In some implementations, multiple sensors 4220 located at various levels 4300 can be connected to a single lighting fixture 1000. For example, a soil sensor can be connected via a long USB extension cable dangled from a USB port 1012B (e.g., an exemplary image is shown in
It should be appreciated that the example node array of
In some implementations, the number of nodes 4200 can be determined by a user-defined density and/or coverage area in the agricultural environment. For example, an IR temperature sensor 4220G can have a finite field of view. An array of integrated sensor assemblies, each corresponding to a node 4200, can thus be installed and spaced apart such that the respective fields of view of the IR temperature sensors 4220G sufficiently overlap to effectively provide sensing coverage for the plants in the environment.
The distribution of nodes 4200 in the node array 4100 can also vary spatially and quantitatively. In some implementations, the nodes 4200 can be uniformly distributed. For example, a uniform array of lighting fixtures 1000 can be deployed with an integrated sensor assembly connected to USB ports on each lighting fixture 1000, as described above. In some implementations, the nodes 4200 distributed in the node array 4100 can be non-uniform. For example, the number of nodes 4200 may vary according to each level 4300 of a plant system where, for example, more nodes 4200 can be used to monitor soil quality than the ambient environment conditions due to variations in coverage by each type of sensor. In another example, an agricultural environment can include different plant species of varying size. The nodes 4200 can be more closely spaced for smaller-sized plants and sparser for larger-sized plants. Additionally, a node 4200 may not include a sensor 4220. Such empty nodes 4200 can be used to define a non-uniform distribution of sensors 4220 with a uniform distribution of nodes 4200. For example, soil quality sensors can occupy every node 4200 at the bottom level 4300 and ambient environment sensors can occupy every other node 4200 at the top level 4300 with empty nodes 4200 in between.
As described above, the node array 4100 can include multiple levels 4300 (e.g., along a z-axis) that correspond to various zones of interest in the controlled growing environment. Zones of interest in a plant system may include a soil level, a plant level, a light canopy level, and an ambient environment level. The soil level can provide data on soil conditions, such as pH value and chemical composition. The plant level can provide data on the leaf temperature or CO2 concentrations near the plant. The light canopy level can provide data on the illumination source, e.g., PPFD, air temperature, relative humidity, or heat dissipation or electrical power for the lighting fixture 1000. The ambient environment level can provide data on air circulation or the temperature of the walls or ceiling of the agricultural environment.
Control Systems
Distributed sensors in the agricultural environment can also be coupled to one or more control systems such that conditions determined by the multisensory imaging system 5050B may be used to adjust the operating parameters of one or more control systems. The control systems can include, but are not limited to, lighting, heating, air flow, hydronics, and humidity conditioning systems. For many agricultural environments, the control systems may be configured to affect growing conditions from a single or few locations in the environment. For example, HVAC systems affecting air flow may be dispersed intermittently along the wall or ceiling in an environment, thus affecting multiple nodes 4200 when operating parameters are changed. In another example, a lighting fixture 1000 can affect growing conditions at nodes 4200 located directly below and near the lighting fixture 1000. Thus, data acquired by one or more sensors 4220 can be used to adjust the control systems such that growing conditions across multiple nodes 4200 are improved or maintained.
Human Machine Interface
In some implementations, the distributed sensor grid 4000 can be connected to an image processor 5000B, as shown in
In some implementations, the HMI 5100 may enable users to select one or more nodes 4200 from an array 4100 in the distributed sensor grid 4000 and display the data collected by these nodes 4200 or conditions 5160B determined by the processor 5000B. To facilitate selection of nodes 4200, the HMI 5100 may include a representation of the agricultural environment. For example,
The HMI 5100 may allow users to display historical data or conditions 5160B determined by the processor 5000B as a function of time. For instance, the environment temperature, relative humidity, electrical power, temperature of a lighting fixture 1000, carbon dioxide concentration, entering water temperature (EWT), leaving water temperature (LWT), and system on a chip (SoC) temperature over a period of several days. Data can be recorded continuously in real-time or incrementally over set time increments (e.g., every 30 minutes, 60 minutes, and 3 hours).
The HMI 5100 may also allow users to adjust control systems (e.g., adjusting the output of a lighting fixture 1000 to simulate a sunrise and sunset). In some implementations, the processor 5000 may automate, at least in part, various controllable conditions based on data from one or more sensors 4420 and user-defined criteria (e.g., set temperature, relative humidity, CO2 concentrations).
As described above, one or more cameras may be coupled to the distributed sensor grid 4000 to record still images or video of a portion of the agricultural environment to allow users to remotely inspect the environment. In some implementations, the selection of a camera can be based on the proximity and field of view of a camera in relation to the nodes 4200 or a control system, e.g., a lighting fixture 1000, selected by a user. Images or video can be acquired on command by a user or recorded on a preset schedule.
An Exemplary Multi-Resolution Multisensory Imaging System
In some implementations, multiple sensors 4220 located at various levels 4300 can be connected to a single lighting fixture 1000. For example, a soil sensor can be connected via a long USB extension cable dangled from a USB port 1012B (e.g., an exemplary image is shown in
In some implementations, the imaging system 1100 may utilize the LED source 400 in the lighting fixture 1000 as an illumination source for acquiring imagery/spectra instead of the LED elements 1142. As described above, the LED source 400 includes one or more LED elements that emit radiation at known wavelengths. In some implementations, each LED element in the LED source 400 may be independently activated similar to the LED elements 1142. Thus, in some implementations, the imaging system 1100 may not include an onboard illumination source instead relying upon other illumination sources in the environment (e.g., the LED source 400).
A feature set of values at each pixel generated by the multisensory imaging system may include sensory data from point sensors 4220 and multispectral imaging data from the imaging engine 1100. The reference condition library 5120 and a machine learning algorithm may be used to determine if one or more reference conditions are present in the multisensory image on a pixel-by-pixel basis.
All parameters, dimensions, materials, and configurations described herein are meant to be exemplary and the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the inventive teachings is/are used. It is to be understood that the foregoing embodiments are presented primarily by way of example and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein.
In addition, any combination of two or more such features, systems, articles, materials, kits, and/or methods, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and arrangement of respective elements of the exemplary implementations without departing from the scope of the present disclosure. The use of a numerical range does not preclude equivalents that fall outside the range that fulfill the same function, in the same way, to produce the same result.
The above-described embodiments can be implemented in multiple ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on a suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.
Further, it should be appreciated that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable or fixed electronic device.
Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format.
Such computers may be interconnected by one or more networks in a suitable form, including a local area network or a wide area network, such as an enterprise network, an intelligent network (IN) or the Internet. Such networks may be based on a suitable technology, may operate according to a suitable protocol, and may include wireless networks, wired networks or fiber optic networks.
The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Some implementations may specifically employ one or more of a particular operating system or platform and a particular programming language and/or scripting tool to facilitate execution.
Also, various inventive concepts may be embodied as one or more methods, of which at least one example has been provided. The acts performed as part of the method may in some instances be ordered in different ways. Accordingly, in some inventive implementations, respective acts of a given method may be performed in an order different than specifically illustrated, which may include performing some acts simultaneously (even if such acts are shown as sequential acts in illustrative embodiments).
All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety.
All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and/or ordinary meanings of the defined terms.
The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”
The phrase “and/or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and/or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and/or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.
As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items in a list, “or” or “and/or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e., “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.
As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B,” or, equivalently “at least one of A and/or B”) can refer, in one embodiment, to at least one, optionally including more than one, A, with no B present (and optionally including elements other than B); in another embodiment, to at least one, optionally including more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, to at least one, optionally including more than one, A, and at least one, optionally including more than one, B (and optionally including other elements); etc.
In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of” and “consisting essentially of” shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.
The present application is a bypass continuation of international application No. PCT/US2020/064382, filed on Dec. 10, 2020, entitled “MULTISENSORY IMAGING METHODS AND APPARATUS FOR CONTROLLED ENVIRONMENT HORTICULTURE USING IRRADIATORS AND CAMERAS AND/OR SENSORS,” which claims priority to U.S. provisional application No. 62/946,407, filed on Dec. 10, 2019, entitled “MULTISPECTRAL IMAGING METHODS AND APPARATUS FOR CONTROLLED ENVIRONMENT HORTICULTURE USING ILLUMINATORS AND CAMERAS AND/OR SENSORS,” each of which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62946407 | Dec 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2020/064382 | Dec 2020 | US |
| Child | 17356429 | US |
