Patent Application
20160162760
1. Technical Field
This description generally relates to the acquisition of material features.
2. Background
The angular dependency and wavelength dependency of incident light that is reflected by the surface of a material are described by the spectral bidirectional reflectance distribution function (BRDF). An extension of the BRDF for modeling the appearance of non-uniform surfaces is given by the bidirectional texture function (BTF), which describes non-local scattering effects such as inter-reflections, subsurface scattering, and shadowing.
In one embodiment, a method comprises obtaining spectral-BRDF material samples, learning feature-vector representations for the spectral-BRDF material samples based on the obtained spectral-BRDF material samples, training classifiers using the learned feature-vector representations, and generating a material classification using the trained classifiers and a new material sample.
In one embodiment, a system comprises one or more computer-readable media; and one or more processors that are coupled to the computer-readable media and that are configured to cause the system to obtain material samples, wherein the material samples include training samples and one or more test samples; learn feature-vector representations of the training samples based on the obtained training samples; and generate a material classification based on the feature-vector representations of the training samples and on the one or more test samples.
In one embodiment, one or more computer-readable media store instructions that, when executed by one or more computing devices, cause the computing devices to perform operations comprising obtaining material samples, wherein the material samples include training samples and one or more test samples; learning feature-vector representations of the training samples based on the obtained training samples; and generating a material classification based on the feature-vector representations of the training samples and on the one or more test samples.
The following description presents certain explanatory embodiments. Other embodiments may include alternatives, equivalents, and modifications. Additionally, the explanatory embodiments may include several novel features, and a particular feature may not be essential to some embodiments of the devices, systems, and methods that are described herein.
A material-classification device 100 includes a computing device, for example, a desktop computer, a laptop computer, a tablet computer, a smartphone, a server, and a personal digital assistant. An illumination-emission device 120 emits lights and may be, for example, a laser or an LED. Also, an illumination-emission device 120 may be tunable (e.g., a tunable laser, a tunable LED) or may be used with a tunable filter, either of which allows the spectrum of light that is emitted by the illumination-emission device 120 to be adjusted. Additionally, an illumination-emission device 120 may have other configurable settings, for example the polarization of emitted light, the filter (e.g., neutral density filters) that the device uses, the intensity of emitted light, and the orientation of the device (when the device is coupled to a movable mount and a motor).
An illumination-detection device 130 detects light and may be, for example, a camera (e.g., an RGB camera, a light-field camera) or a photometer. An illumination-detection device 130 may include a tunable sensor or a tunable filter, either of which allows the spectrum of light that is detected by the illumination-detection device 130 to be adjusted. Also, an illumination-detection device 130 may have other configurable settings, for example dynamic range, shutter speed, aperture, signal gain (ISO), polarization, the filter used by the device, focal plane, and orientation (when the device is coupled to a movable mount and a motor). An illumination-detection device 130 generates one or more images 131 or other illumination-indicating signals based on the light that it detects.
The images 131 may include a BRDF-image stack, a hyperspectral-image stack (e.g., a hyperspectral data cube), or a combined BRDF-hyperspectral-image stack. Also, an image slice refers to one image in an image stack. And an image patch refers to a subset of an image that extends across the image stack. Furthermore, depending on the embodiment, a sample can be an entire image, an image stack, an image slice, or an image patch. Therefore, a material sample can be an entire image of a material, an image stack of a material, an image slice of a material, or an image patch of a material.
The flow starts in block 200, where one or more training images of materials are obtained. Next, in block 210, feature-vector representations are generated for the materials based on the one or more training images, for example based on image slices or image patches from the training images. Block 210 includes generating a neural-network model (e.g., a reduced-Boltzmann-machine model), which includes hidden-layer units and neural-network-model parameters, based on the training images (e.g., on image slices, on image patches). The flow then moves to block 220, where classifiers are generated based on the feature-vector representations. Then, in block 230, one or more test images of an unknown material are obtained.
Next, in block 240, missing data for the test images is reconstructed based on the neural-network model. For example, in an industrial setting, illumination-emission devices and illumination-detection devices can malfunction. Furthermore, even if a malfunction is detected immediately, repairing the malfunction may require several minutes. During this time, the process in a factory, for example, may continue to run because halting an entire process for a small malfunction is not always practical. Thus, in a setting that includes a large number of illumination-detection systems and imaging setups, ensuring that the system is constantly running can be crucial.
And if a part of the system malfunctions, for example if one or more illumination-emission devices fail, then the corresponding images (or, in some embodiments, image slices) may be dark. Consequently, the feature vectors calculated for those images may be meaningless. Additionally, the images (or image slices) could be confused with dark spectral images (or image slices), which could arise when a material is not responsive to a specific wavelength of light. Therefore, some images (or image slices) may be dark even when all of the components of the system are functioning properly.
An illustration is shown in
Therefore, referring again to
Based on the images 531, the one or more material-classification devices 500 generate a neural-network model 510, which includes generating neural-network-model parameters 515 and hidden-layer units 513. For example, some embodiments extract vectorized patches from the images 531 and use the vectorized patches as visible-layer units of a visible layer to generate the neural-network model 510. Then, based on the hidden-layer units 513, the one or more material-classification devices 500 generate feature-vector representations 501 of the one or more materials that compose the first object 550A and the one or more materials that compose the second object 550B. The feature-vector representations 501 may be generated using, for example, k-means clustering or mixture models (e.g., Gaussian Mixture Models). Also, the feature-vector representations 501 may include the hidden-layer units 513 themselves, for example in embodiments that use patches as visible-layer units, or may include histograms of the hidden-layer units 513, for example in embodiments that use image slices as visible-layer units. Additionally, the feature-vector representations 501 may indicate the pixel intensities of their respective images.
Furthermore, based on the feature-vector representations 501, the one or more material-classification devices 500 generate classifiers 502 for the one or more materials that compose the first object 550A and the one or more materials that compose the second object 550B. Because the materials are known, the one or more material-classification devices 500 store the classifiers 502 and the feature-vector representations 501 in association with their respective materials.
The flow then proceeds to block 650, where feature-vector representations are generated for the materials based on one or more of the hidden-layer units and the visible-layer units. Finally, some embodiments proceed to block 660, where classifiers are generated based on the feature-vector representations.
The one or more material-classification devices 700 obtain a neural-network model 710, which includes neural-network-model parameters and hidden-layer units, and generate test hidden-layer units 716 based on the images 731 and on the neural-network model 710. The one or more material-classification devices 700 then calculate visible-layer units 717 based on the test hidden-layer units 716 and on the neural-network-model parameters. Next, the one or more material-classification devices 700 recalculate hidden-layer units 718 based on the calculated visible-layer units 717 and on the neural-network-model parameters. Also, the one or more material-classification devices 700 generate a feature-vector representation 701 for the object 750 based on the recalculated hidden-layer units 718.
Based on the feature-vector representations 701, the one or more material-classification devices 700 generate a material classification 703 for the object 750. In some embodiments, the one or more material-classification device 700 obtain one or more classifiers 702 and generate the material classification 703 based on the one or more classifiers 702 and the feature-vector representation 701. In some embodiments, the one or more material-classification devices 700 obtain the training feature-vector representations (e.g., the feature-vector representations 501 in
For example, some embodiments use distance-based classification. The distance between each test feature-vector representation 701 and all the training feature-vector representations is computed using the chi-squared distance, which is a distance between histograms. The training feature-vector representations are ranked by order of increasing distance from the test feature-vector representations. Some embodiments classify the material based on the top three feature-vector representations: the material classification 703 is taken to be either that of a material that has multiple feature-vector representations in the top three or, if there is no repetition, the material that has the top-ranked feature-vector representation.
Next, in block 840, recomputed hidden-layer units are generated based on the computed visible-layer units and on the neural-network-model parameters. Then in block 850, a feature-vector representation for the object is generated based on the recomputed hidden-layer units. The flow then moves to block 860, where trained classifiers are obtained. Finally, in block 870, the materials of the object are classified based on the trained classifiers and on the feature-vector representation.
Some embodiments omit block 860 and 870 and generate a material classification based on the feature-vector representation for the object and on obtained feature-vector representations for different materials. Some of these embodiments use distance-based classification (e.g., chi-squared distance) or an SVM (Support Vector Machine) classifier to generate the material classification for the object.
The material-classification device 1000 includes one or more processors (CPUs) 1004, I/O interfaces 1005, and storage/memory 1006. Also, the components of the material-classification device 1000 communicate via a bus. The CPUs 1004 includes one or more central processing units, which include microprocessors (e.g., a single core microprocessor, a multi-core microprocessor) or other circuits, and the CPU 1004 is configured to read and perform computer-executable instructions, such as instructions in storage, in memory, or in a module. The I/O interfaces 1005 include communication interfaces to input and output devices, which may include a keyboard, a display, a mouse, a printing device, a touch screen, a light pen, an optical-storage device, a scanner, a microphone, a camera, a drive, and a network (either wired or wireless).
The storage/memory 1006 includes one or more computer-readable or computer-writable media, for example a computer-readable storage medium. A computer-readable storage medium, in contrast to a mere transitory, propagating signal, includes a tangible article of manufacture, for example a magnetic disk (e.g., a floppy disk, a hard disk), an optical disc (e.g., a CD, a DVD, a Blu-ray), a magneto-optical disk, magnetic tape, and semiconductor memory (e.g., a non-volatile memory card, flash memory, a solid-state drive, SRAM, DRAM, EPROM, EEPROM). The storage/memory 1006 can store computer-readable data or computer-executable instructions.
The material-classification device 1000 also includes a training module 1007, a testing module 1008, and a classification module 1009. A module includes logic, computer-readable data, or computer-executable instructions, and a module may be implemented in software (e.g., Assembly, C, C++, C#, Java, BASIC, Perl, Visual Basic), hardware (e.g., customized circuitry), or a combination of software and hardware. For example, a module may be a self-contained hardware or software component that interfaces with a larger system, a packaged functional hardware unit designed for use with other components, or a part of a program that usually performs a particular function of related functions. In some embodiments, the devices in the system include additional or fewer modules, the modules are combined into fewer modules, or the modules are divided into more modules.
The training module 1007 includes instructions that, when executed, or circuits that, when activated, cause the material-classification device 1000 to obtain one or more training images of one or more materials, extract visual features from the training images, generate a neural-network model based on the visual features, and generate feature-vector representations for the one or more materials based on the neural-network model. Also, in some embodiments, the training module 1007 includes instructions that, when executed, or circuits that, when activated, cause the material-classification device 1000 to train classifiers based on the feature-vector representations.
The testing module 1008 includes instructions that, when executed, or circuits that, when activated, cause the material-classification device 1000 to obtain a neural-network model, obtain one or more test images, compute hidden-layer units for the one or more test images based on the test images and on the neural-network model, compute visible-layer units based on the neural-network model and on the hidden-layer units for the one or more test images, recompute hidden-layer units for the one or more test images based on the computed visible-layer units and on the neural-network model, and generate a feature vector representation for the one or more test images based on the recomputed hidden-layer units.
The classification module 1009 includes instructions that, when executed, or circuits that, when activated, cause the material-classification device 1000 to classify the materials of an object based on the feature-vector representation of the object and the trained feature-vector representations of one or more materials, or to classify the materials of an object based on the feature-vector representation of the object and on trained classifiers of one or more materials.
The illumination-emission device 1020 includes one or more processors (CPUs) 1021, I/O interfaces 1022, storage/memory 1023, an emission-control module 1024, and at least one illumination-emission component (e.g., a laser, an LED light) that is configured to emit illumination. The emission-control module 1024 includes instructions that, when executed, or circuits that, when activated, cause the illumination-emission device 1020 to configure the illumination-emission component according to received illumination-emission-settings values and to activate and deactivate the illumination-emission component, for example according to received emission-timing-setting values.
The illumination-detection device 1030 includes one or more processors (CPUs) 1032, I/O interfaces 1033, storage/memory 1034, a detection-control module 1035, and at least one illumination-detection component (e.g., a CMOS sensor, a CCD sensor) that is configured to detect illumination. The detection-control module 1035 includes instructions that, when executed, or circuits that, when activated, cause the illumination-detection device 1030 to configure the illumination-detection component according to received illumination-detection-settings values and to activate and deactivate the illumination-detection component, for example according to received detection-timing-setting values.
For example, in one embodiment, the system of
For each material set, the system obtained 4 material samples, for a total of 8 material samples. The system used different combinations or folds of training and test data, such that for each fold the test data contained one material sample from each category. In total, the system used 16 folds of training and testing data, where 75% of the material samples were used for training and 25% were used for testing. Therefore, each fold of a training set had 6 material samples, while a test set had 2 material samples. The classification accuracy was then computed for each set by either (i) taking the average over the accuracies computed for each of the test sets of the 16 folds or (ii) taking the median over the accuracies computed for each of the test sets of the 16 folds.
Table 2 shows classification results over all the 25 sets when the mean and median were calculated over the 25 means and the 25 medians.
In one embodiment, a system was designed to assume that there was missing data at test time, although the training data was complete. In this particular embodiment, the system was designed to assume that clusters 6 and 7 in the RIT imaging dome were broken at test time. Therefore the pixel values of the image slices corresponding to these LED clusters were set to 0. The system used the same metal sets that are shown in Table 1 and used the same setup.
The system used the following two types of feature vectors: (i) a 1×150 mean feature vector taken over 150 image slices of an image stack for one material, and (ii) a 1×450 msHOR feature vector. The system also trained an SVM classifier using a training set that included a complete set of images, and the system tested the classifier using test images with missing images. The mean and median classification accuracies taken over the 25 metals sets were lower in the case of missing data in the test images for both types of feature vectors. Table 3 shows the results for the mean classification accuracies when the two types of feature vectors are considered: the mean feature vector computed as the concatenation of the means over the image pixel-intensity values, and the msHOR feature vectors computed as the concatenation of the HOR feature vectors over all 150 image slices.
Table 4 shows the results for the median classification accuracies when the two types of feature vectors were considered: the mean feature vector computed as the concatenation of the means over the image pixel-intensity values, and the msHOR feature vector computed as the concatenation of the HOR feature vectors over all 150 image slices.
In one embodiments, a system performed classification using the same training and test data as above, both when the images were complete and when a few images were missing. The feature-vector representations were based on the histograms of hidden-layer units. Table 5 shows the results when the images were missing. Note that these results are identical to the ones in Table 2. This implies that the feature vectors are robust to the missing data phenomenon.
The above-described devices and systems can be implemented, at least in part, by providing one or more computer-readable media that contain computer-executable instructions for realizing the above-described operations to one or more computing devices that are configured to read and execute the computer-executable instructions. The systems or devices perform the operations of the above-described embodiments when executing the computer-executable instructions. Also, an operating system on the one or more systems or devices may implement at least some of the operations of the above-described embodiments.
Any applicable computer-readable medium (e.g., a magnetic disk (including a floppy disk, a hard disk), an optical disc (including a CD, a DVD, a Blu-ray disc), a magneto-optical disk, a magnetic tape, and semiconductor memory (including flash memory, DRAM, SRAM, a solid state drive, EPROM, EEPROM)) can be employed as a computer-readable medium for the computer-executable instructions. The computer-executable instructions may be stored on a computer-readable storage medium that is provided on a function-extension board inserted into a device or on a function-extension unit connected to the device, and a CPU provided on the function-extension board or unit may implement at least some of the operations of the above-described embodiments.
The scope of the claims is not limited to the above-described embodiments and includes various modifications and equivalent arrangements. Also, as used herein, the conjunction “or” generally refers to an inclusive “or,” though “or” may refer to an exclusive “or” if expressly indicated or if the context indicates that the “or” must be an exclusive “or.”
