Systems and methods for surface modeling using polarization cues

Description

FIELD

Aspects of embodiments of the present disclosure relate to the field of computer vision and the modeling of surfaces of objects using machine vision.

BACKGROUND

Large scale surface modeling is often desirable in manufacturing for a variety of reasons. One area of use is in the manufacturing of automobiles and automotive parts, where surface modeling using computer vision or machine vision provides methods for automated inspection of the scanned surfaces, which may improve efficiency and result in cost reduction in manufacturing.

Large scale surface modeling may also be applied in other contexts, such as laboratory work and inspection of individual workpieces outside of large scale manufacturing.

SUMMARY

Aspects of embodiments of the present disclosure relate to surface modeling by using light polarization (e.g., the rotation of light waves) to provide additional channels of information to the process of characterizing the surfaces of objects. Aspects of embodiments of the present disclosure may be applied in scenarios such as manufacturing, where surface characterization is used to perform object inspection as a component of a quality assurance process, such as detecting defective goods produced on a manufacturing line and removing or repairing those defective objects.

According to one embodiment of the present disclosure, a computer-implemented method for surface modeling includes: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames being captured at different polarizations by a polarization camera including a polarizing filter; extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and detecting a surface characteristic of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces.

The one or more first tensors in the one or more polarization representation spaces may include: a degree of linear polarization (DOLP) image in a DOLP representation space; and an angle of linear polarization (AOLP) image in an AOLP representation space.

The one or more first tensors may further include one or more non-polarization tensors in one or more non-polarization representation spaces, and the one or more non-polarization tensors may include one or more intensity images in intensity representation space.

The one or more intensity images may include: a first color intensity image; a second color intensity image; and a third color intensity image.

The surface characteristic may include a detection of a defect in the surface of the physical object.

The detecting the surface characteristic may include: loading a stored model corresponding to a location of the surface of the physical object; and computing the surface characteristic in accordance with the stored model and the one or more first tensors in the one or more polarization representation spaces.

The stored model may include one or more reference tensors in the one or more polarization representation spaces, and the computing the surface characteristic may include computing a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.

The difference may be computed using a Fresnel distance.

The stored model may include a reference three-dimensional mesh, and the computing the surface characteristic may include: computing a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and computing a difference between the three-dimensional point cloud and the reference three-dimensional mesh.

The stored model may include a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.

The trained statistical model may include an anomaly detection model.

The trained statistical model may include a convolutional neural network trained to detect defects in the surface of the physical object.

The trained statistical model may include a trained classifier trained to detect defects.

According to one embodiment of the present disclosure, a system for surface modeling includes: a polarization camera including a polarizing filter, the polarization camera being configured to capture polarization raw frames at different polarizations; and a processing system including a processor and memory storing instructions that, when executed by the processor, cause the processor to: receive one or more polarization raw frames of a surface of a physical object, the polarization raw frames corresponding to different polarizations of light; extract one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and detect a surface characteristic of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces.

The one or more intensity images may include: a first color intensity image; a second color intensity image; and a third color intensity image.

The surface characteristic may include a detection of a defect in the surface of the physical object.

The memory may further store instructions that, when executed by the processor, cause the processor to detect the surface characteristic by: loading a stored model corresponding to a location of the surface of the physical object; and computing the surface characteristic in accordance with the stored model and the one or more first tensors in the one or more polarization representation spaces.

The stored model may include one or more reference tensors in the one or more polarization representation spaces, and the memory may further store instructions that, when executed by the processor, cause the processor to compute the surface characteristic by computing a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.

The difference may be computed using a Fresnel distance.

The stored model may include a reference three-dimensional mesh, and the memory may further store instructions that, when executed by the processor, cause the processor to compute the surface characteristic by: computing a three-dimensional point cloud of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces; and computing a difference between the three-dimensional point cloud and the reference three-dimensional mesh.

The trained statistical model may include an anomaly detection model.

The trained statistical model may include a convolutional neural network trained to detect defects in the surface of the physical object.

The trained statistical model may include a trained classifier trained to detect defects.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, together with the specification, illustrate exemplary embodiments of the present invention, and, together with the description, serve to explain the principles of the present invention.

FIG. 1A is a schematic depiction of surfaces of an object (e.g., an automobile) under inspection by a surface characterization system according to one embodiment of the present disclosure.

FIG. 1B is a schematic block diagram of a surface characterization system according to one embodiment of the present invention.

FIG. 2A is an image or intensity image of a scene with one real transparent ball placed on top of a printout of photograph depicting another scene containing two transparent balls (“spoofs”) and some background clutter.

FIG. 2B depicts the intensity image of FIG. 2A with an overlaid segmentation mask as computed by a Mask Region-based Convolutional Neural Network (Mask R-CNN) identifying instances of transparent balls, where the real transparent ball is correctly identified as an instance, and the two spoofs are incorrectly identified as instances.

FIG. 2C is an angle of polarization image computed from polarization raw frames captured of the scene according to one embodiment of the present invention.

FIG. 3 is a high-level depiction of the interaction of light with transparent objects and non-transparent (e.g., diffuse and/or reflective) objects.

FIG. 4 is a graph of the energy of light that is transmitted versus reflected over a range of incident angles to a surface having a refractive index of approximately 1.5.

FIG. 5 is a block diagram of processing circuit 100 for computing surface characterization outputs based on polarization data according to one embodiment of the present invention.

FIG. 6 is a flowchart of a method for performing surface characterization based on input images to compute a surface characterization output according to one embodiment of the present invention.

FIG. 7A is a block diagram of a feature extractor according to one embodiment of the present invention.

FIG. 7B is a flowchart depicting a method according to one embodiment of the present invention for extracting features from polarization raw frames.

FIG. 8A is a block diagram of a predictor according to one embodiment of the present invention.

FIG. 8B is a flowchart depicting a method according to one embodiment of the present invention for detecting characteristics of surfaces of objects.

DETAILED DESCRIPTION

In the following detailed description, only certain exemplary embodiments of the present invention are shown and described, by way of illustration. As those skilled in the art would recognize, the invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Like reference numerals designate like elements throughout the specification.

As used herein, the term “surface modeling” refers to capturing information about the surfaces of real-world objects, such as the three-dimensional shape of the surface, and may also include capturing color (or “texture”) information about the surface and other information about the reflectivity of the surface (e.g., a bidirectional reflectance distribution function or BRDF).

Surface profile examination is important in analyzing the intrinsic shape and curvature properties or characteristics of surfaces. Surface modeling of real-world objects has applications in many areas in which the characterization of surfaces is desired. For example, in manufacturing, surface modeling may be used to perform inspection of the objects produced through the manufacturing process, thereby enabling the detection of defects in the objects (or manufactured goods or workpieces) and removal of those defective objects from the manufacturing stream. One area of use is in the manufacturing of automobiles and automotive parts, such as in the automatic detection of defective automotive parts, where a computer vision or machine vision system captures images of the automotive parts (e.g., using one or more cameras) and generates a classification result and/or other detection information regarding the quality of the part, such as whether a window is scratched or whether a door panel is dented. Applying surface modeling techniques using computer vision to perform automated inspection of the scanned surfaces improves the efficiency and reduces costs in manufacturing, such as by detecting errors early in the manufacturing or assembly process.

Computer vision and machine vision techniques enable rapid and contactless surface modeling, in contrast to, for example, contact three-dimensional (3-D) scanners that probe a subject through physical touch. However, comparative computer vision techniques, whether performed passively (e.g., without additional illumination) or actively (e.g., with an active illumination device, which may emit structure light), may fail to reliably certain classes of surface characteristics that may be termed “optically challenging.” These may be circumstances where the color of the defect is very similar to the background color of the surface on which the defect appears. For example, defects such as scratches in a glass window or in the clear coat layers of a glossy paint and shallow dents in painted or unpainted metal surfaces may often be difficult to see in a standard color image of a surface, because the color (or texture) variation due to these defects may be relatively small. In other words, the contrast between the color of the defect and the color of the non-defective (or “clean”) surfaces may be relatively small, such as where a dent in a painted door panel has the same color as in the undented portion

Accordingly, some aspects of embodiments of the present disclosure relate to detecting defects in objects based on polarization features of objects, as computed based on raw polarization frames captured of objects under inspection using one or more polarization cameras (e.g., cameras that include a polarizing filter in the optical path). Polarization-enhanced imaging can provide, in some embodiments, order of magnitude improvements to the characterization of the shapes of surface, including the accuracy of the detected direction of surface normals. Aesthetically smooth surfaces cannot have bumps or dents, which are essentially variations in local curvature which in turn are defined by their surface normal representations. Accordingly, some embodiments of the present disclosure can be applied to smoothness detection and shape fidelity in high precision manufacturing of industrial parts. One use case involves the inspection of manufactured parts before they leave the assembly line for delivery to the end customer. In many manufacturing systems, manufactured parts come off the assembly line on a conveyor system (e.g., on a conveyor belt) at high rates and, for efficiency of throughput, require that the inspection happen while the part is still moving, and with very little time between parts.

As such, some aspects of embodiments of the present disclosure relate to systems and methods for surface characterization, including through the capture of polarization raw frames of the surfaces to be characterized, and computing characterizations, such as detecting defects in the surface, based on those polarization raw frames.

FIG. 1A is a schematic depiction of surfaces of an object (e.g., an automobile) under inspection by a surface characterization system according to one embodiment of the present disclosure. In the arrangement shown in FIG. 1A, an object under inspection 1 may be within a scene or environment. For example, in the context of a factory or other manufacturing plant, the object under inspection 1 may be located on an assembly line and may be in motion on a conveyor system 40 such as a conveyor belt or an overhead conveyor (e.g., an overhead chain conveyor). The object under inspection 1 may have one or more surfaces (labeled in FIG. 1A as surface 2, surface 3, and surface 4), which are imaged by one or more polarization cameras 10. The polarization cameras 10 may be mounted on a mount, where the mount may be a movable mount, such as on an end-effector of a robotic arm 32 or may be a fixed mount, such as fixed on a gantry 34 above the conveyor system or as part of the conveyor system. The polarization cameras 10 capture polarization raw frames (images) 18 of various surfaces 2, 3, and 4 of the object under inspection 1, where each polarization camera 10 includes a polarizing filter in its optical path.

FIG. 1B is a schematic block diagram of a surface characterization system according to one embodiment of the present invention. In particular, FIG. 1B depicts one of the polarization cameras 10 configured to image surface 2 of the object under inspection 1. In the embodiment shown in FIG. 1B, a polarization camera 10 has a lens 12 with a field of view, where the lens 12 and the camera 10 are oriented such that the field of view encompasses the surface under inspection (e.g., surface under inspection 2 of the object under inspection 1). The lens 12 is configured to direct light (e.g., focus light) from the scene (e.g., from the surface under inspection) onto a light sensitive medium such as an image sensor 14 (e.g., a complementary metal oxide semiconductor (CMOS) image sensor or charge-coupled device (CCD) image sensor).

The polarization camera 10 further includes a polarizer or polarizing filter or polarization mask 16 placed in the optical path between the scene 1 and the image sensor 14. According to various embodiments of the present disclosure, the polarizer or polarization mask 16 is configured to enable the polarization camera 10 to capture images of the scene 1 with the polarizer set at various specified angles (e.g., at 45° rotations or at 60° rotations or at non-uniformly spaced rotations).

As one example, FIG. 1B depicts an embodiment where the polarization mask 16 is a polarization mosaic aligned with the pixel grid of the image sensor 14 in a manner similar to a red-green-blue (RGB) color filter (e.g., a Bayer filter) of a color camera. In a manner similar to how a color filter mosaic filters incoming light based on wavelength such that each pixel in the image sensor 14 receives light in a particular portion of the spectrum (e.g., red, green, or blue) in accordance with the pattern of color filters of the mosaic, a polarization mask 16 using a polarization mosaic filters light based on linear polarization such that different pixels receive light at different angles of linear polarization (e.g., at 0°, 45°, 90°, and 135°, or at 0°, 60° degrees, and 120°). Accordingly, the polarization camera 10 using a polarization mask 16 such as that shown in FIG. 1 is capable of concurrently or simultaneously capturing light at four different linear polarizations. One example of a polarization camera is the Blackfly® S Polarization Camera produced by FLIR® Systems, Inc. of Wilsonville, Oreg.

While the above description relates to some possible implementations of a polarization camera using a polarization mosaic, embodiments of the present disclosure are not limited thereto and encompass other types of polarization cameras that are capable of capturing images at multiple different polarizations. For example, the polarization mask 16 may have fewer than four polarizations or more than four different polarizations, or may have polarizations at different angles than those stated above (e.g., at angles of polarization of: 0°, 60°, and 120° or at angles of polarization of 0°, 30°, 60°, 90°, 120°, and 150°). As another example, the polarization mask 16 may be implemented using an electronically controlled polarization mask, such as an electro-optic modulator (e.g., may include a liquid crystal layer), where the polarization angles of the individual pixels of the mask may be independently controlled, such that different portions of the image sensor 14 receive light having different polarizations. As another example, the electro-optic modulator may be configured to transmit light of different linear polarizations when capturing different frames, e.g., so that the camera captures images with the entirety of the polarization mask set to, sequentially, to different linear polarizer angles (e.g., sequentially set to: 0 degrees; 45 degrees; 90 degrees; or 135 degrees). As another example, the polarization mask 16 may include a polarizing filter that rotates mechanically, such that different polarization raw frames are captured by the polarization camera 10 with the polarizing filter mechanically rotated with respect to the lens 12 to transmit light at different angles of polarization to image sensor 14. Furthermore, while the above examples relate to the use of a linear polarizing filter, embodiments of the present disclosure are not limited thereto and also include the use of polarization cameras that include circular polarizing filters (e.g., linear polarizing filters with a quarter wave plate). Accordingly, in various embodiments of the present disclosure, a polarization camera uses a polarizing filter to capture multiple polarization raw frames at different polarizations of light, such as different linear polarization angles and different circular polarizations (e.g., handedness).

As a result, the polarization camera 10 captures multiple input images 18 (or polarization raw frames) of the scene including the surface under inspection 2 of the object under inspection 1. In some embodiments, each of the polarization raw frames 18 corresponds to an image taken behind a polarization filter or polarizer at a different angle of polarization ϕ_pol(e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). Each of the polarization raw frames 18 is captured from substantially the same pose with respect to the scene 1 (e.g., the images captured with the polarization filter at 0 degrees, 45 degrees, 90 degrees, or 135 degrees are all captured by a same polarization camera 100 located at a same location and orientation), as opposed to capturing the polarization raw frames from disparate locations and orientations with respect to the scene. The polarization camera 10 may be configured to detect light in a variety of different portions of the electromagnetic spectrum, such as the human-visible portion of the electromagnetic spectrum, red, green, and blue portions of the human-visible spectrum, as well as invisible portions of the electromagnetic spectrum such as infrared and ultraviolet.

In some embodiments of the present disclosure, such as some of the embodiments described above, the different polarization raw frames are captured by a same polarization camera 10 and therefore may be captured from substantially the same pose (e.g., position and orientation) with respect to the scene 1. However, embodiments of the present disclosure are not limited thereto. For example, a polarization camera 10 may move with respect to the scene 1 between different polarization raw frames (e.g., when different raw polarization raw frames corresponding to different angles of polarization are captured at different times, such as in the case of a mechanically rotating polarizing filter), either because the polarization camera 10 has moved or because object 1 has moved (e.g., if the object is on a moving conveyor system). In some embodiments, different polarization cameras capture images of the object at different times, but from substantially the same pose with respect to the object (e.g., different cameras capturing images of the same surface of the object at different points in the conveyor system). Accordingly, in some embodiments of the present disclosure different polarization raw frames are captured with the polarization camera 10 at different poses or the same relative pose with respect to the object under inspection 1 and/or the surface under inspection 2.

The polarization raw frames 18 are supplied to a processing circuit 100, described in more detail below, which computes a characterization output 20 based on the polarization raw frames 18. In the embodiment shown in FIG. 1B, the characterization output 20, includes a region 21 of the image of the surface 2 in which a defect is detected (e.g., a dent in a door of an automobile).

FIGS. 2A, 2B, 2C, and 2D provide background for illustrating the segmentation maps computed by a comparative approach and semantic segmentation or instance segmentation based on polarization raw frames according to embodiments of the present disclosure. In more detail, FIG. 2A is an image or intensity image of a scene with one real transparent ball placed on top of a printout of photograph depicting another scene containing two transparent balls (“spoofs”) and some background clutter. FIG. 2B depicts a segmentation mask as computed by a Mask Region-based Convolutional Neural Network (Mask R-CNN) identifying instances of transparent balls overlaid on the intensity image of FIG. 2A using different patterns of lines, where the real transparent ball is correctly identified as an instance, and the two spoofs are incorrectly identified as instances. In other words, the Mask R-CNN algorithm has been fooled into labeling the two spoof transparent balls as instances of actual transparent balls in the scene.

FIG. 2C is an angle of linear polarization (AOLP) image computed from polarization raw frames captured of the scene according to one embodiment of the present invention. As shown in FIG. 2C, transparent objects have a very unique texture in polarization space such as the AOLP domain, where there is a geometry-dependent signature on edges and a distinct or unique or particular pattern that arises on the surfaces of transparent objects in the angle of linear polarization. In other words, the intrinsic texture of the transparent object (e.g., as opposed to extrinsic texture adopted from the background surfaces visible through the transparent object) is more visible in the angle of polarization image of FIG. 2C than it is in the intensity image of FIG. 2A.

FIG. 2D depicts the intensity image of FIG. 2A with an overlaid segmentation mask as computed using polarization data in accordance with an embodiment of the present invention, where the real transparent ball is correctly identified as an instance using an overlaid pattern of lines and the two spoofs are correctly excluded as instances (e.g., in contrast to FIG. 2B, FIG. 2D does not include overlaid patterns of lines over the two spoofs). While FIGS. 2A, 2B, 2C, and 2D illustrate an example relating to detecting a real transparent object in the presence of spoof transparent objects, embodiments of the present disclosure are not limited thereto and may also be applied to other optically challenging objects, such as transparent, translucent, and non-matte or non-Lambertian objects, as well as non-reflective (e.g., matte black objects) and multipath inducing objects.

Accordingly, some aspects of embodiments of the present disclosure relate to extracting, from the polarization raw frames, tensors in representation space (or first tensors in first representation spaces, such as polarization feature maps) to be supplied as input to surface characterization algorithms or other computer vision algorithms. These first tensors in first representation space may include polarization feature maps that encode information relating to the polarization of light received from the scene such as the AOLP image shown in FIG. 2C, degree of linear polarization (DOLP) feature maps, and the like (e.g., other combinations from Stokes vectors or transformations of individual ones of the polarization raw frames). In some embodiments, these polarization feature maps are used together with non-polarization feature maps (e.g., intensity images such as the image shown in FIG. 2A) to provide additional channels of information for use by semantic segmentation algorithms.

While embodiments of the present invention are not limited to use with particular surface characterization algorithms, some aspects of embodiments of the present invention relate to deep learning frameworks for polarization-based surface characterization of transparent objects (e.g., glass windows of vehicles and transparent glossy layers of paints) or other optically challenging objects (e.g., transparent, translucent, non-Lambertian, multipath inducing objects, and non-reflective (e.g., very dark) objects), where these frameworks may be referred to as Polarized Convolutional Neural Networks (Polarized CNNs). This Polarized CNN framework includes a backbone that is suitable for processing the particular texture of polarization and can be coupled with other computer vision architectures such as Mask R-CNN (e.g., to form a Polarized Mask R-CNN architecture) to produce a solution for accurate and robust characterization of transparent objects and other optically challenging objects. Furthermore, this approach may be applied to scenes with a mix of transparent and non-transparent (e.g., opaque objects) and can be used to characterize transparent, translucent, non-Lambertian, multipath inducing, dark, and opaque surfaces of the object or objects under inspection.

Polarization Feature Representation Spaces

Some aspects of embodiments of the present disclosure relate to systems and methods for extracting features from polarization raw frames in operation 650, where these extracted features are used in operation 690 in the robust detection of optically challenging characteristics in the surfaces of objects. In contrast, comparative techniques relying on intensity images alone may fail to detect these optically challenging features or surfaces (e.g., comparing the intensity image of FIG. 2A with the AOLP image of FIG. 2C, discussed above). The term “first tensors” in “first representation spaces” will be used herein to refer to features computed from (e.g., extracted from) polarization raw frames 18 captured by a polarization camera, where these first representation spaces include at least polarization feature spaces (e.g., feature spaces such as AOLP and DOLP that contain information about the polarization of the light detected by the image sensor) and may also include non-polarization feature spaces (e.g., feature spaces that do not require information regarding the polarization of light reaching the image sensor, such as images computed based solely on intensity images captured without any polarizing filters).

The interaction between light and transparent objects is rich and complex, but the material of an object determines its transparency under visible light. For many transparent household objects, the majority of visible light passes straight through and a small portion (˜4% to ˜8%, depending on the refractive index) is reflected. This is because light in the visible portion of the spectrum has insufficient energy to excite atoms in the transparent object. As a result, the texture (e.g., appearance) of objects behind the transparent object (or visible through the transparent object) dominate the appearance of the transparent object. For example, when looking at a transparent glass cup or tumbler on a table, the appearance of the objects on the other side of the tumbler (e.g., the surface of the table) generally dominate what is seen through the cup. This property leads to some difficulties when attempting to detect surface characteristics of transparent objects such as glass windows and glossy, transparent layers of paint, based on intensity images alone:

FIG. 3 is a high-level depiction of the interaction of light with transparent objects and non-transparent (e.g., diffuse and/or reflective) objects. As shown in FIG. 3, a polarization camera 10 captures polarization raw frames of a scene that includes a transparent object 302 in front of an opaque background object 303. A light ray 310 hitting the image sensor 14 of the polarization camera 10 contains polarization information from both the transparent object 302 and the background object 303. The small fraction of reflected light 312 from the transparent object 302 is heavily polarized, and thus has a large impact on the polarization measurement, in contrast to the light 313 reflected off the background object 303 and passing through the transparent object 302.

Similarly, a light ray hitting the surface of an object may interact with the shape of the surface in various ways. For example, a surface with a glossy paint may behave substantially similarly to a transparent object in front of an opaque object as shown in FIG. 3, where interactions between the light ray and a transparent or translucent layer (or clear coat layer) of the glossy paint causes the light reflecting off of the surface to be polarized based on the characteristics of the transparent or translucent layer (e.g., based on the thickness and surface normals of the layer), which are encoded in the light ray hitting the image sensor. Similarly, as discussed in more detail below with respect to shape from polarization (SfP) theory, variations in the shape of the surface (e.g., direction of the surface normals) may cause significant changes in the polarization of light reflected by the surface of the object. For example, smooth surfaces may generally exhibit the same polarization characteristics throughout, but a scratch or a dent in the surface changes the direction of the surface normals in those areas, and light hitting scratches or dents may be polarized, attenuated, or reflected in ways different than in other portions of the surface of the object. Models of the interactions between light and matter generally consider three fundamentals: geometry, lighting, and material. Geometry is based on the shape of the material. Lighting includes the direction and color of the lighting. Material can be parameterized by the refractive index or angular reflection/transmission of light. This angular reflection is known as a bi-directional reflectance distribution function (BRDF), although other functional forms may more accurately represent certain scenarios. For example, the bidirectional subsurface scattering distribution function (BSSRDF) would be more accurate in the context of materials that exhibit subsurface scattering (e.g. marble or wax).

A light ray 310 hitting the image sensor 16 of a polarization camera 10 has three measurable components: the intensity of light (intensity image/I), the percentage or proportion of light that is linearly polarized (degree of linear polarization/DOLP/ρ), and the direction of that linear polarization (angle of linear polarization/AOLP/ϕ). These properties encode information about the surface curvature and material of the object being imaged, which can be used by the predictor 800 to detect transparent objects, as described in more detail below. In some embodiments, the predictor 800 can detect other optically challenging objects based on similar polarization properties of light passing through translucent objects and/or light interacting with multipath inducing objects or by non-reflective objects (e.g., matte black objects).

Therefore, some aspects of embodiments of the present invention relate to using a feature extractor 700 to compute first tensors in one or more first representation spaces, which may include derived feature maps based on the intensity I, the DOLP ρ, and the AOLP ϕ. The feature extractor 700 may generally extract information into first representation spaces (or first feature spaces) which include polarization representation spaces (or polarization feature spaces) such as “polarization images,” in other words, images that are extracted based on the polarization raw frames that would not otherwise be computable from intensity images (e.g., images captured by a camera that did not include a polarizing filter or other mechanism for detecting the polarization of light reaching its image sensor), where these polarization images may include DOLP ρ images (in DOLP representation space or feature space), AOLP ϕ images (in AOLP representation space or feature space), other combinations of the polarization raw frames as computed from Stokes vectors, as well as other images (or more generally first tensors or first feature tensors) of information computed from polarization raw frames. The first representation spaces may include non-polarization representation spaces such as the intensity I representation space.

Measuring intensity I, DOLP ρ, and AOLP ϕ at each pixel requires 3 or more polarization raw frames of a scene taken behind polarizing filters (or polarizers) at different angles, ϕ_pol(e.g., because there are three unknown values to be determined: intensity I, DOLP ρ, and AOLP ϕ. For example, the FLIR® Blackfly® S Polarization Camera described above captures polarization raw frames with polarization angles ϕ_polat 0 degrees, 45 degrees, 90 degrees, or 135 degrees, thereby producing four polarization raw frames I_ϕ_pol, denoted herein as I₀, I₄₅, I₉₀, and I₁₃₅.

The relationship between I_ϕ_poland intensity I, DOLP ρ, and AOLP ϕ at each pixel can be expressed as:

I_ϕ_pol=I(1+ρ cos(2((ϕ−ϕ_pol) (1)

Accordingly, with four different polarization raw frames I_pol(I₀, I₄₅, I₉₀, and I₁₃₅), a system of four equations can be used to solve for the intensity I, DOLP ρ, and AOLP ϕ.

Shape from Polarization (SfP) theory (see, e.g., Gary A Atkinson and Edwin R Hancock. Recovery of surface orientation from diffuse polarization. IEEE transactions on image processing, 15(6):1653-1664, 2006.) states that the relationship between the refractive index (n), azimuth angle (θ_a) and zenith angle (θ_z) of the surface normal of an object and the ϕ and ρ components of the light ray coming from that object follow the following characteristics when diffuse reflection is dominant:

$\begin{matrix} ρ = \frac{{(n - \frac{1}{n})}^{2} \sin^{2} (θ_{z})}{2 + 2 n^{2} - {(n + \frac{1}{n})}^{2} \sin^{2} θ_{z} + 4 \cos θ_{z} \sqrt{n^{2} - \sin^{2} θ_{z}}} & (2) \\ ϕ = θ_{a} & (3) \end{matrix}$

and when the specular reflection is dominant:

$\begin{matrix} ρ = \frac{2 \sin^{2} θ_{z} \cos θ_{z} \sqrt{n^{2} - \sin^{2} θ_{z}}}{n^{2} - \sin^{2} θ_{z} - n^{2} \sin^{2} θ_{z} + 2 \sin^{4} θ_{z}} & (4) \\ ϕ = θ_{a} - \frac{π}{2} & (5) \end{matrix}$

Note that in both cases ρ increases exponentially as Oz increases and if the refractive index is the same, specular reflection is much more polarized than diffuse reflection.

Accordingly, some aspects of embodiments of the present disclosure relate to applying SfP theory to detect the shapes of surfaces (e.g., the orientation of surfaces) based on the raw polarization frames 18 of the surfaces. This approach enables the shapes of objects to be characterized without the use of other computer vision techniques for determining the shapes of objects, such as time-of-flight (ToF) depth sensing and/or stereo vision techniques, although embodiments of the present disclosure may be used in conjunction with such techniques.

More formally, aspects of embodiments of the present disclosure relate to computing first tensors 50 in first representation spaces, including extracting first tensors in polarization representation spaces such as forming polarization images (or extracting derived polarization feature maps) in operation 650 based on polarization raw frames captured by a polarization camera 10.

Light rays coming from a transparent objects have two components: a reflected portion including reflected intensity I_r, reflected DOLP ρ_r, and reflected AOLP ϕ_rand the refracted portion including refracted intensity I_t, refracted DOLP ρ_t, and refracted AOLP ϕ_t. The intensity of a single pixel in the resulting image can be written as:

I=I_r+I_t (6)

When a polarizing filter having a linear polarization angle of ϕ_polis placed in front of the camera, the value at a given pixel is:

I_ϕ_pol=I_r(1+ρ_rcos(2(ϕ_r−ϕ_pol))+I_t(1+ρ_tcos(2(ϕ_t−ϕ_pol) (7)

Solving the above expression for the values of a pixel in a DOLP ρ image and a pixel in an AOLP ϕ image in terms of I_r, ρ_r, ϕ_r, I_t, ρ_t, and ϕ_t:

$\begin{matrix} ρ = \frac{\sqrt{{(I_{r} ρ_{r})}^{2} + {(I_{t} ρ_{t})}^{2} + 2 I_{t} ρ_{t} I_{r} ρ_{r} \cos (2 (ϕ_{r} - ϕ_{t}))}}{I_{r} + I_{t}} & (8) \\ ϕ = \arctan (\frac{I_{r} ρ_{r} \sin (2 (ϕ_{r} - ϕ_{t}))}{I_{r} ρ_{t} + I_{r} ρ_{r} \cos (2 (ϕ_{r} - ϕ_{t}))}) + ϕ_{r} & (9) \end{matrix}$

Accordingly, equations (7), (8), and (9), above, provide a model for forming first tensors 50 in first representation spaces that include an intensity image I, a DOLP image ρ, and an AOLP image ϕ according to one embodiment of the present disclosure, where the use of polarization images or tensor in polarization representation spaces (including DOLP image ρ and an AOLP image ϕ based on equations (8) and (9)) enables the reliable detection of optically challenging surface characteristics of objects that are generally not detectable by comparative systems that use only intensity I images as input.

In more detail, first tensors in polarization representation spaces (among the derived feature maps 50) such as the polarization images DOLP ρ and AOLP ϕ can reveal surface characteristics of objects that might otherwise appear textureless in an intensity I domain. A transparent object may have a texture that is invisible in the intensity domain I because this intensity is strictly dependent on the ratio of I_r/I_t(see equation (6)). Unlike opaque objects where I_t=0, transparent objects transmit most of the incident light and only reflect a small portion of this incident light. As another example, thin or small deviations in the shape of an otherwise smooth surface (or smooth portions in an otherwise rough surface) may be substantially invisible or have low contrast in the intensity I domain (e.g., a domain in which polarization of light is not taken into account), but may be very visible or may have high contrast in a polarization representation space such as DOLP ρ or AOLP ϕ.

As such, one exemplary method to acquire surface topography is to use polarization cues in conjunction with geometric regularization. The Fresnel equations relate the AOLP ϕ and the DOLP ρ with surface normals. These equations can be useful for anomaly detection by exploiting what is known as polarization patterns of the surface. A polarization pattern is a tensor of size [M, N, K] where M and N are horizontal and vertical pixel dimensions, respectively, and where K is the polarization data channel, which can vary in size. For example, if circular polarization is ignored and only linear polarization is considered, then K would be equal to two, because linear polarization has both an angle and a degree of polarization (AOLP ϕ and DOLP ρ). Analogous to a Moire pattern, in some embodiments of the present disclosure, the feature extraction module 700 extracts a polarization pattern in polarization representation spaces (e.g., AOLP space and DOLP space). In the example characterization output 20 shown in FIG. 1A and FIG. 1B shown above, the horizontal and vertical dimensions correspond to the lateral field of view of a narrow strip or patch of the surface 2 captured by the polarization camera 10. However, this is one exemplary case: in various embodiments, the strip or patch of the surface may be vertical (e.g., much taller than wide), horizontal (e.g., much wider than tall), or have a more conventional field of view (FoV) that tends closer toward a square (e.g., a 4:3 ratio or 16:9 ratio of width to height).

While the preceding discussion provides specific examples of polarization representation spaces based on linear polarization in the case of using a polarization camera having one or more linear polarizing filters to capture polarization raw frames corresponding to different angles of linear polarization and to compute tensors in linear polarization representation spaces such as DOLP and AOLP, embodiments of the present disclosure are not limited thereto. For example, in some embodiments of the present disclosure, a polarization camera includes one or more circular polarizing filters configured to pass only circularly polarized light, and where polarization patterns or first tensors in circular polarization representation space are further extracted from the polarization raw frames. In some embodiments, these additional tensors in circular polarization representation space are used alone, and in other embodiments they are used together with the tensors in linear polarization representation spaces such as AOLP and DOLP. For example, a polarization pattern including tensors in polarization representation spaces may include tensors in circular polarization space, AOLP, and DOLP, where the polarization pattern may have dimensions [M,N,K], where K is three to further include the tensor in circular polarization representation space.

FIG. 4 is a graph of the energy of light that is transmitted versus reflected over a range of incident angles to a surface having a refractive index of approximately 1.5. As shown in FIG. 4, the slopes of the transmitted energy (shown in FIG. 4 with a solid line) and reflected energy (shown in FIG. 4 with a dotted line) lines are relatively small at low incident angles (e.g., at angles closer to perpendicular to the plane of the surface). As such, small differences in the angle of the surface may be difficult to detect (low contrast) in the polarization pattern when the angle of incidence is low (e.g., close to perpendicular to the surface, in other words, close to the surface normal). On the other hand, the slope of the reflected energy increases from flat, as the angle of incidence increases, and the slope of the transmitted energy decreases from flat (to have a larger absolute value) as the angle of incidence increases. In the example shown in FIG. 4 with an index of refraction of 1.5, the slopes of both lines are substantially steeper beginning at an incident angle of around 60°, and their slopes are very steep at an incident angle of around 80°. The particular shapes of the curves may change for different materials in accordance with the refractive index of the material. Therefore, capturing images of surfaces under inspection at incident angles corresponding to steeper portions of the curves (e.g., angles close to parallel to the surface, such as around 80° in the case of a refractive index of 1.5, as shown in FIG. 4) can improve the contrast and detectability of variations in the surface shapes in the polarization raw frames 18 and may improve the detectability of such features in tensors in polarization representation spaces, because small changes in incident angle (due to the small changes in the surface normal) can cause large changes in the captured polarization raw frames.

Accordingly, some aspects of embodiments of the present disclosure relate to supplying first tensors in the first representation spaces (e.g., including feature maps in polarization representation spaces) extracted from polarization raw frames as inputs to a predictor for computing or detecting surface characteristics of transparent objects and/or other optically challenging surface characteristics of objects under inspection. These first tensors may include derived feature maps which may include an intensity feature map I, a degree of linear polarization (DOLP) ρ feature map, and an angle of linear polarization (AOLP) ϕ feature map, and where the DOLP ρ feature map and the AOLP ϕ feature map are examples of polarization feature maps or tensors in polarization representation spaces, in reference to feature maps that encode information regarding the polarization of light detected by a polarization camera. In some embodiments, the feature maps or tensors in polarization representation spaces are supplied as input to, for example, detection algorithms that make use of SfP theory to characterize the shape of surfaces of objects imaged by the polarization cameras 10.

Surface Characterization Based on Polarization Features

As shown above in FIGS. 1A and 1B, aspects of embodiments of the present invention relate to systems and methods for performing surface characterization of objects under inspection by capturing images of surfaces of objects 1 using one or more polarization cameras 10 that capture polarization raw frames 18 that are analyzed by a processing system or processing circuit 100. The characterization of the surfaces may include detecting surface characteristics that are optically challenging, e.g., surface characteristics that may be difficult or impossible to detect using comparative computer vision or machine vision techniques that do not use polarization information. While some aspects of embodiments of the present disclosure relate to surface characteristics that correspond to defects in manufactured products (e.g., defects such as cracks, tears, uneven application of paints or dyes, the presence of surface contaminants, unintentional surface irregularities or other geometric deviations from reference models, and the like), embodiments of the present disclosure are not limited thereto and may be applied to detecting other surface characteristics such as detecting the locations boundaries between different types of materials, measuring the uniformity of the refractive index of a material across an area, characterizing geometry of surface treatments applied to portions of materials (e.g., the etching of materials and/or the depositing of materials onto surfaces), and the like.

FIG. 5 is a block diagram of processing circuit 100 for computing surface characterization outputs based on polarization data according to one embodiment of the present invention. FIG. 6 is a flowchart of a method 600 for performing surface characterization based on input images to compute a surface characterization output according to one embodiment of the present invention.

According to various embodiments of the present disclosure, the processing circuit 100 is implemented using one or more electronic circuits configured to perform various operations as described in more detail below. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator (e.g., a vector processor, which may include vector arithmetic logic units configured efficiently perform operations common to neural networks, such dot products and softmax), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or the like. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented in program instructions that are stored in a non-volatile computer readable memory where, when executed by the electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or combinations thereof), perform the operations described herein to compute a characterization output 20 from input polarization raw frames 18. The operations performed by the processing circuit 100 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, or the like) or may be allocated between multiple electronic circuits (e.g., multiple GPUs or a CPU in conjunction with a GPU). The multiple electronic circuits may be local to one another (e.g., located on a same die, located within a same package, or located within a same embedded device or computer system) and/or may be remote from one other (e.g., in communication over a network such as a local personal area network such as Bluetooth®, over a local area network such as a local wired and/or wireless network, and/or over wide area network such as the internet, such a case where some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits operating to implement the processing circuit 100 may be referred to herein as a computer or a computer system, which may include memory storing instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.

As shown in FIG. 5, in some embodiments, a processing circuit 100 includes a feature extractor or feature extraction system 700 and a predictor 800 (e.g., a classical computer vision prediction algorithm and/or a trained statistical model such as a trained neural network) configured to compute a prediction output 20 (e.g., a statistical prediction) regarding surface characteristics of objects based on the output of the feature extraction system 700. While some embodiments of the present disclosure are described herein in the context of a surface characterization system for detecting defects in the surfaces of manufactured objects, where those surface defects may be optically challenging to detect, embodiments of the present disclosure are not limited thereto. For example, some aspects of embodiments of the present disclosure may be applied to techniques for characterizing the surfaces of objects made of materials or have surface characteristics that are optically challenging to detect, such as surfaces of translucent objects, multipath inducing objects, objects that are not entirely or substantially matte or Lambertian, and/or very dark objects. These optically challenging objects include objects and surface characteristics thereof that are difficult to resolve or detect through the use of images that are capture by camera systems that are not sensitive to the polarization of light (e.g., based on images captured by cameras without a polarizing filter in the optical path or where different images do not capture images based on different polarization angles). For example, these surface characteristics may have surface appearances or colors that are very similar to the surfaces on which the characteristics appear (e.g., dents have the same color as the underlying material and scratches on transparent materials such as glass may also be substantially transparent). In addition, while embodiments of the present disclosure are described herein in the context of detecting optically challenging surface characteristics, embodiments of the present disclosure are not limited to detecting only optically challenging surface defects. For example, in some embodiments, a predictor 800 is configured (e.g., a statistical model is trained using training data) to detect both surface characteristics that are optically challenging as well as surface characteristics that are robustly detectable without using polarization information.

Polarization may be used to detect surface characteristics or features that would otherwise be optically challenging when using intensity information (e.g., color intensity information) alone. For example, polarization information can detect changes in geometry and changes in material in the surfaces of objects. The changes in material (or material changes), such as boundaries between different types of materials (e.g., a black metallic object on a black road or a colorless liquid on a surface may both be substantially invisible in color space, but would both have corresponding polarization signatures in polarization space), may be more visible in polarization space because differences in the refractive indexes of the different materials cause changes in the polarization of the light. Likewise, differences in the specularity of various materials cause different changes in the polarization phase angle of rotation, also leading to detectable features in polarization space that might otherwise be optically challenging to detect without using a polarizing filter. Accordingly, this causes contrast to appear in images or tensors in polarization representation spaces, where corresponding regions of tensors computed in intensity space (e.g., color representation spaces that do not account for the polarization of light) may fail to capture these surface characteristics (e.g., where these surface characteristics have low contrast or may be invisible in these spaces). Examples of optically challenging surface characteristics include: the particular shapes of the surfaces (e.g., degree of smoothness and deviations from ideal or acceptable physical design tolerances for the surfaces); surface roughness and shapes of the surface roughness patterns (e.g., intentional etchings, scratches, and edges in the surfaces of transparent objects and machined parts), burrs and flash at the edges of machined parts and molded parts; and the like. Polarization would also be useful to detect objects with identical colors, but differing material properties, such as scattering or refractive index.

As shown in FIG. 6 and referring, for example, to FIG. 1B, in operation 610 the processing circuit 100 captures polarization raw frames 18 of a surface 2 of an object under inspection 1. For example, in some embodiments, the processing circuit 100 controls one or more polarization cameras 10 to capture polarization raw frames 18 depicting a particular surface 2 of the object. In various embodiments of the present disclosure, the capture of particular surfaces of the object under inspection may be triggered using one or more detection systems, such as a mechanical switch trigger (e.g. when a portion of the object or a conveyor system closes an electronic switch to signal the current location of the object), a laser trigger (e.g., when a portion of the object 1 blocks a laser beam from reaching a detector), or an optical trigger (e.g., a camera system detects the presence of the object at a particular location).

Referring back to FIG. 1A, a polarization-enhanced imaging system or surface characterization system according to embodiments of the present disclosure may use polarization cameras 10 mounted either on a gantry located around the conveyor belt or on the end-effector of robotic arms, which can be used to provide patch-based images of the object 1 (e.g., images or patches or strips of surfaces of the object 1) as the object 1 moves on the conveyor belt. In some embodiments of the present disclosure, polarization cameras 10 attached to the movable mounts are automatically repositioned by the system to place the polarization cameras 10 in poses such that the incident angle of the light on the surface is in the steeper or higher contrast portion of the curves shown in FIG. 4 (e.g., based on the general orientations of the surface under inspection and the light sources in the scene). In some embodiments of the present disclosure, illumination sources (e.g., running lights or flashes) may also be placed in fixed locations or attached to the movable mounts (e.g., either rigidly attached to a corresponding polarization camera or attached to an independently movable moveable mount) to illuminate the surfaces of the object with light at an incident angle that makes the surface shape features of the object more easily detectable (e.g., at high incident angles).

Accordingly, in some embodiments of the present disclosure, capturing the polarization raw frames 18 of a surface 2 of the object under inspection 1 in operation 610 includes moving polarization cameras 10 and/or illumination sources to poses with respect to the surface 2 under inspection in accordance with particular characteristics of the surface 2 to be characterized. For example, in some embodiments, this involves automatically positioning the polarization cameras 10 and/or illumination sources such that light from the illumination sources strikes the surface 2 at a high incident angle (e.g., around 80 degrees). In some embodiments of the present disclosure, the particular positions at which high incident angles may be feasible will vary based on the particular shapes of the surfaces to be inspected (e.g., the design of a door of a car may include different portions with significantly different surface normals, such as an indentation at the door handle, edges where the door meets the window, and indentations in the main surface of the door for style and/or aerodynamics).

In some embodiments of the present disclosure, the processing circuit 100 loads a profile associated with the type or class of object under inspection, where the profile includes a collection of one or more poses for the polarization camera 10 to be moved to in relation to the object under inspection 1. Different types or classes of objects having different shapes may be associated with different profiles, while manufactured objects of the same type or class are expected to have the same shape. (For example, different models of vehicles may have different shapes, and these different models of vehicles may be mixed in an assembly line. Accordingly, the processing circuit 100 may select, from a collection of different profiles, a profile corresponding to the type of vehicle currently under inspection.) Accordingly, the polarization camera 10 may be automatically moved through a sequence of poses stored in the profile to capture polarization raw frames 18 of the surfaces of the object under inspection 1.

In the embodiment shown in FIGS. 5 and 6, in operation 650, the feature extraction system 700 of the processing circuit 100 extracts one or more first feature maps 50 in one or more first representation spaces (including polarization images or polarization feature maps in various polarization representation spaces) from the input polarization raw frames 18 of a scene.

FIG. 7A is a block diagram of a feature extractor 700 according to one embodiment of the present invention. FIG. 7B is a flowchart depicting a method according to one embodiment of the present invention for extracting features from polarization raw frames. In the embodiment shown in FIG. 7A, the feature extractor 700 includes an intensity extractor 720 configured to extract an intensity image I 52 in an intensity representation space (e.g., in accordance with equation (7), as one example of a non-polarization representation space) and polarization feature extractors 730 configured to extract features in one or more polarization representation spaces. In some embodiments of the present disclosure, the intensity extractor 720 is omitted and the feature extractor does not extract an intensity image I 52.

As shown in FIG. 7B, the extraction of polarization images in operation 650 may include extracting, in operation 651, a first tensor in a first polarization representation space from the polarization raw frames from a first Stokes vector. In operation 652, the feature extractor 700 further extracts a second tensor in a second polarization representation space from the polarization raw frames. For example, the polarization feature extractors 730 may include a DOLP extractor 740 configured to extract a DOLP ρ image 54 (e.g., a first polarization image or a first tensor in accordance with equation (8) with DOLP as the first polarization representation space) and an AOLP extractor 760 configured to extract an AOLP ϕ image 56 (e.g., a second polarization image or a second tensor in accordance with equation (9), with AOLP as the second polarization representation space) from the supplied polarization raw frames 18. In addition, in various embodiments, the feature extraction system 700 extracts two or more different tensors (e.g., n different tensors) in two or more representation spaces (e.g., n representation spaces), where the n-th tensor is extracted in operation 614. As discussed above, in some embodiments of the present disclosure, the polarization feature extractors 730 extracts polarization features in polarization representation spaces including both linear polarization representation spaces (e.g., tensors in the aforementioned AOLP and DOLP representation spaces extracted from polarization raw frames captured with a linear polarizing filter) and circular polarization representation spaces (e.g., tensors extracted from polarization raw frames captured with a circular polarizing filter). In various embodiments, the representation spaces include, but are not limited to, polarization representation spaces.

The polarization representation spaces may include combinations of polarization raw frames in accordance with Stokes vectors. As further examples, the polarization representations may include modifications or transformations of polarization raw frames in accordance with one or more image processing filters (e.g., a filter to increase image contrast or a denoising filter). The feature maps 52, 54, and 56 in first polarization representation spaces may then be supplied to a predictor 800 for detecting surface characteristics based on the feature maps 50.

While FIG. 7B illustrates a case where two or more different tensors are extracted from the polarization raw frames 18 in more than two different representation spaces, embodiments of the present disclosure are not limited thereto. For example, in some embodiments of the present disclosure, exactly one tensor in a polarization representation space is extracted from the polarization raw frames 18. For example, one polarization representation space of raw frames is AOLP ϕ and another is DOLP ρ (e.g., in some applications, AOLP may be sufficient for detecting surface characteristics of transparent objects or surface characteristics of other optically challenging objects such as translucent, non-Lambertian, multipath inducing, and/or non-reflective objects).

Accordingly, extracting features such as polarization feature maps or polarization images from polarization raw frames 18 produces first tensors 50 from which optically challenging surface characteristics may be detected from images of surfaces of objects under inspection. In some embodiments, the first tensors extracted by the feature extractor 700 may be explicitly derived features (e.g., hand crafted by a human designer) that relate to underlying physical phenomena that may be exhibited in the polarization raw frames (e.g., the calculation of AOLP and DOLP images in linear polarization spaces and the calculation of tensors in circular polarization spaces, as discussed above). In some additional embodiments of the present disclosure, the feature extractor 700 extracts other non-polarization feature maps or non-polarization images, such as intensity maps for different colors of light (e.g., red, green, and blue light) and transformations of the intensity maps (e.g., applying image processing filters to the intensity maps). In some embodiments of the present disclosure the feature extractor 700 may be configured to extract one or more features that are automatically learned (e.g., features that are not manually specified by a human) through an end-to-end supervised training process based on labeled training data. In some embodiments, these learned feature extractors may include deep convolutional neural networks, which may be used in conjunction with traditional computer vision filters (e.g., a Haar wavelet transform, a Canny edge detector, and the like).

Surface Characterization Based on Tensors in Representation Spaces Including Polarization Representation Spaces

The feature maps in first representation space 50 (including polarization images) extracted by the feature extraction system 700 are provided as input to the predictor 800 of the processing circuit 100, which implements one or more prediction models to compute, in operation 690, a surface characterization output 20.

In the case where the predictor 800 is a defect detection system, the prediction may be an image 20 (e.g., an intensity image) of the surface 2, where a portion of the image is marked 21 or highlighted as containing a defect. In some embodiments, the output of the defect detection system is a segmentation map, where each pixel may be associated with one or more confidences that the pixel corresponds to a location of various possible classes (or types) of surface characteristics (e.g., defects) that may be found in objects that the surface characterization system is trained to inspect, or a confidence that the pixel corresponds to an anomalous condition in the image of the surface of the object under inspection. In the case where the predictor is a classification system, the prediction may include a plurality of classes and corresponding confidences that the image depicts an instance of each of the classes (e.g. that the image depicts various types of defects or different types of surface characteristics such as smooth glass, etched glass, scratched glass, and the like). In the case where the predictor 800 is a classical computer vision prediction algorithm, the predictor may compute a detection result (e.g., detect defects by comparing the extracted feature maps in first representation space to model feature maps in the first representation space or identify edges or regions with sharp or discontinuous changes in the feature map in areas that are expected to be smooth).

In the embodiment shown in FIG. 5, the predictor 800 implements a defect detection system and computes, in operation 690, a surface characterization output 20 that includes a location of a detected defect, which is computed based on the extracted first tensors 50 in first representation spaces, extracted from the input polarization raw frames 18. As noted above, the feature extraction system 700 and the predictor 800 are implemented using one or more electronic circuits that are configured to perform their operations, as described in more detail below.

According to various embodiments of the present disclosure, the surface 2 of the object 1 as imaged by the one or more polarization cameras 10 is characterized in accordance with a model associated with the surface. The particular details of the surface characterization performed by a surface characterization system according to embodiments of the present invention depend on the particular application and the surfaces being characterized.

Continuing the above example of the detection of defects on the surfaces of an automobile, different types of defects may appear on different surfaces of the automobile, due to the locations and methods of manufacturing the various parts and due to the types of materials used in the different parts. For example, painted metal door panels may exhibit different types of defects (e.g., scratches, dents) than glass windows (e.g., scratches, chips, and cracks), which may exhibit defects that are different from those found in plastic components (e.g., headlight covers, which may also show scratches, chips, and cracks, but may also contain expected and intentional surface irregularities, including such as surface ridges and bumps and ejector pin marks).

As another example, in a machined, metal part, some surfaces may be expected to be smooth and glossy, while other surfaces may be expected to be rough or to have particular physical patterns (e.g., patterns of grooves, bumps, or random textures), where different surfaces of the machined part may have different tolerances.

FIG. 8A is a block diagram of a predictor according to one embodiment of the present invention. As shown in FIG. 8A, a predictor 800 receives input tensors in first representation spaces 50. The predictor 800 may include a collection of models 810 associated with different types of surfaces that are expected to be analyzed by the surface characterization system. In the embodiment shown in FIG. 8A, the predictor 800 has access to m different models (e.g., different models stored in the memory of the processing circuit 100). For example, a first model 811 may be associated with main surfaces of door panels, a second model 812, may be associated with the handle portion of a door panel, and an m-th model 814 may be associated with a tail light.

FIG. 8B is a flowchart depicting a method 690 according to one embodiment of the present invention for detecting characteristics of surfaces of objects. In operation 691, the processing system 100 selects a model, from among the collection of models 810, that corresponds to the current surface. In some embodiments, the particular model is selected based on metadata stored in the profile associated with the object under inspection 1 and associated with the particular pose at which the polarization raw frames 18 were captured by the polarization camera 10.

In some embodiments of the present disclosure the orientation of the objects under inspection is consistent from one object to the next. For example, in the case of automobile manufacturing, each assembled automobile may move along the conveyor system with its nose leading (e.g. as opposed to some moving with the drivers' side leading and some with the rear of the vehicle leading). Accordingly, images of different surfaces of the object under inspection 1 may be reliably captured based on known information about the position of the automobile on the conveyor system and its speed. For example, a camera located at a particular height on the drivers' side of the automobile may expect to image a particular portion of the bumper, the fender, the wheel wells, the drivers' side door, the quarter panel, and the rear bumper of the car. Based on the speed of the conveyor system and a triggering time at which the automobile enters the field of view of the surface characterization system, various surfaces of the automobile will be expected to be imaged at different times, in accordance with a profile associated with the type of the object (e.g., the type, class, or model of the car).

In some embodiments, the orientations of the objects under inspection may be inconsistent, and therefore a separate registration process may be employed to determine which surfaces are being imaged by the polarization cameras 10. In these embodiments, the profile may include a three-dimensional (3-D) model of the object under inspection (e.g., a computer aided design or CAD model of the physical object or three-dimensional mesh or point cloud model). Accordingly, in some embodiments, a simultaneous location and mapping (SLAM) algorithm is applied to determine which portions of object under inspection are being imaged by the polarization cameras 10 and to use the determined locations to identify corresponding locations on the 3-D model, thereby enabling a determination of which surfaces of the 3-D model were imaged by the polarization cameras 10. For example, keypoint detection algorithms may be used to detect unique parts of the object, and the keypoints are used to match the orientation of the 3-D model to the orientation of the physical object under inspection 1.

As such, in some embodiments of the present disclosure, a surface registration module 820 of the prediction system 800 registers the polarization raw frames 18 captured by the polarization cameras (and/or the tensors in representation spaces 50) with particular portions of the object under inspection based on a profile associated with the object to select a model associated with the current surface imaged by the polarization raw frames 18 from a collection of models 810.

In operation 693, the processing system applies the selected model using the surface analyzer 830 to compute the surface characterization output 20 for the current surface. Details of the various types of models and the particular operations performed by the surface analyzer 830 based on these different types of models according to various embodiments of the present disclosure will be described in more detail below.

Surface Characterization Through Comparison with Design Models and Representative Models

In some embodiments of the present disclosure, the stored models include feature maps in representation space as computed from representative models (e.g., design models) of the objects under inspection, and the surface analyzer compares the feature maps computed from the captured polarization raw frames 18 against the stored representative (e.g., ideal) feature maps in the same representation space.

For example, as noted above, in some embodiments of the present disclosure, the representation spaces include a degree of linear polarization (DOLP) ρ and an angle of linear polarization (AOLP) ϕ. In some such embodiments, the models 810 include reference 2-D and/or 3-D models (e.g., CAD models) of the surfaces, which have their intrinsic surface normals. These intrinsic surface reference models are sometimes referred to as design surface normals and are the design targets of the surface (e.g., ideal shapes of the surface), and therefore these represent the ground truth for the patch under inspection (e.g., the patch of the surface imaged by the set of polarization raw frames 18).

In such embodiments, the feature extraction system 700 extracts surface normals using shape from polarization (SfP), and these surface normal are aligned, by the surface registration module 820, with the reference 2-D and/or 3-D models (e.g., the CAD models) of the corresponding part of surface.

In this embodiment, the surface analyzer 830 performs a comparison between the surface normals represented in the tensors in representation space 50 computed from the polarization raw frames 18 and the design surface normals from a corresponding one of the models 810 to find the regions of discrepancy whereby the different areas are identified and flagged. For example, portions of the tensors in representation spaces 50 computed from the raw polarization frames 18 that differ from the corresponding portions of the design surface normals (in the same representation spaces as the tensors 50) by more than a threshold amount are marked as discrepancies or potential defects, while other portions that differ by less than the threshold are marked as clean (e.g., not defective). In various embodiments of the present disclosure, this threshold may be set based on, for example, designed tolerances for the surface under inspection and the sensitivity of the system (e.g., in accordance with noise levels in the system, such as sensor noise in the image sensor 14 of the polarization camera 10.

In addition, given that the regions of interest have both the computed surface normals and the 3-D coordinates of the surface from the design target loaded from the selected model from the models 810, in some embodiments the surface analyzer 830 converts the regions into 3-D point clouds representing the shape of the imaged surface (e.g., using shape from polarization equations), and the surface analyzer 830 performs further inspection and analysis on the generated 3-D point clouds, such as by comparing the shapes of the 3-D point clouds to the shapes of the corresponding surfaces in the reference 3-D models. The comparison may include iteratively reorienting the point clouds to minimize the distance between the points in the point cloud and the surface of the reference 3-D models, where points of the point cloud that are more than a threshold distance away from the surface of the reference 3-D model regions of the surface under inspection that deviate from the reference model and that may correspond to geometric defects (e.g., dents, burrs, or other surface irregularities).

As another example, manufactured parts that meet the same tolerance will have substantially the same polarization patterns under similar lighting (e.g., the same polarization patterns, with variations due to the manufacturing tolerances). The polarization pattern of an ideal or expected or reference part will be referred to as a template polarization pattern or a reference tensor (which would correspond to the model selected from the set of models 810). In these embodiments, the feature extraction system 700 extracts a measured polarization pattern for a surface of an object under inspection (e.g., measured tensors in first representation spaces the AOLP and DOLP feature maps described above). If the surface of the object contained an anomaly, such as a micro-dent in the surface, this anomaly would appear in the measured polarization pattern, thereby resulting in its classification as an anomalous polarization pattern (or having a region containing an anomaly, such as region 21 shown in FIG. 1B) that differs from the template polarization pattern or reference tensors in the first representation spaces. On the other hand, a defect-free surface would generate a measured polarization pattern that matched (within tolerances) the template polarization pattern or reference tensors (e.g., when the measured polarization pattern matches, then it is classified as a clean polarization pattern).

Some aspects of embodiments of the present disclosure relate to mathematical operations for comparing the template polarization pattern and measured polarization patterns. In some embodiments, a subtraction or arithmetic difference between the template and anomalous polarization patterns is computed to compare the patterns. However, as shown in FIG. 4, the Fresnel equations model the non-linear relationship between incident angle and energy transmitted and energy reflected, where the shapes of the curves shift in accordance with refractive index (FIG. 4 shows example curves for a refractive index of 1.5). This non-linear change in energy reflected for similar changes in surface normals at different incident angles may make it difficult to perform comparisons between polarization patterns (e.g., comparing a template polarization pattern against a measured polarization pattern). For example, a 1 degree change in incident angle in the neighborhood of 60 degrees (e.g., an average incident angle of 60 degrees and a surface normal variation that caused a 0.5 degree change in incident angle to 60.5 degrees) would have a larger change in energy reflected than a similar change in the neighborhood of 10 degrees (e.g., an average incident angle of 0 degrees and a surface normal variation that caused a 0.5 degree change in incident angle to 0.5 degrees). In other words, these embodiments would use a linear metric for comparing a non-linear phenomenon, which may cause detectability problems in flatter neighborhoods of the curve (e.g., portions of the curve with a smaller first derivative) or may cause saturation or overflow of the signal in steeper neighborhoods of the curve (e.g., portions of the curve with a larger first derivative).

As such, some aspects of embodiments of the present disclosure relate to the use of Fresnel subtraction to compute a Fresnel distance for comparing a template polarization pattern and a measured polarization pattern in a manner that that accounts for the non-linear relationship between the incident angle and the energy reflected or transmitted. Accordingly, Fresnel subtraction according to some aspects of embodiments of the present disclosure is a non-linear operator that admits linear comparison of surface normals. In effect, Fresnel subtraction linearizes the curve shown in FIG. 4, enabling a relative micro-surface deviation of 30 degrees to be represented with a consistent anomaly score (e.g., an anomaly score computed in accordance with a Fresnel distance), whether the original orientation was at 0° or 60° (e.g., average incident angle over the surface). In other words, a Fresnel distance is computed using a Fresnel subtraction in accordance with embodiments of present disclosure, where the Fresnel distance between two polarization patterns is substantially independent of the original orientation of the surface (e.g., substantially independent of the average incident angle over the surface). In some embodiments of the present disclosure, a Fresnel subtraction function is parametrically learned using the pattern matching technique of symbolic regression. In some embodiments of the present disclosure, a Fresnel subtraction function is numerically approximated based on the known Fresnel equations in accordance with the refractive index of the material and the orientation of the surface, such as by dividing the measured reflected light by the percentage of energy reflected at the approximate incident angle of light on the surface (e.g., the average incident angle over a substantially planar local patch of the surface), based on an assumption that the variation of the surface normal is small enough to be within a substantially or sufficiently linear neighborhood of the curve. In some embodiments of the present disclosure, closed form equations are derived based on prior knowledge of the material properties, such as the refractive index of the material.

Because the Fresnel equations are refractive index dependent, Fresnel Subtraction is also dependent on the refractive index of the material (e.g., the shapes of the curves shown in FIG. 4 shift in accordance with the refractive index). Manufactured parts may have different refractive indexes in different patches (e.g., on different surfaces). In some embodiments of the present disclosure, a standard refractive index is selected based on balancing the sensitivity needs with respect to different surfaces of the object in accordance with the application (e.g., the contact surfaces of manufactured parts may be more important than non-contact surfaces of those manufactured parts, and therefore a refractive index closer to that of the contact surfaces may be selected). For example, the standard refractive index may be set to 1.5 and assumed to be sufficiently close.

In some embodiments of the present disclosure, local calibration with the design surface normals is performed to determine a locally smooth refractive index for each patch, thereby enabling a higher precision Fresnel Subtraction that is tailored for each patch. In some embodiments, local calibration is performed by assuming that the refractive index is a scalar constant that does not vary across different pixels and using information from different pixels to estimate the value of the refractive index for a given material. In some embodiments, local calibration is performed by estimating refractive index values using the techniques described in the “refractive distortion” section of Kadambi, Achuta, et al. “Polarized 3d: High-quality depth sensing with polarization cues.” Proceedings of the IEEE International Conference on Computer Vision. 2015.

As such, some aspects of embodiments of the present disclosure relate to detecting defects by comparing measured feature maps or tensors extracted from polarization raw frames captured of an object under inspection against reference tensors or reference feature maps or template feature maps corresponding to reference or template objects (e.g., based on ideal surfaces from the design, such as a CAD model, or based on measurements of a known good object).

Surface Feature Detection Using Anomaly Detection Algorithms

In some embodiments of the present disclosure, surface features are detected using anomaly detection. For example, in some circumstances, some significant variation may be expected from one instance of an object under inspection to the next. For example, manufacturing processes may cause irregular and non-uniform variations in the polarization patterns exhibited by materials. While these variations may be within manufacturing tolerances, these variations may not be aligned with particular physical locations relative to the object as a whole. For example, glass window may exhibit some inconsistent polarization patterns from one window to the next, in accordance with the cooling process of the particular sheet of glass. However, the inconsistency in the polarization patterns may make it difficult to detect defects. For example, if a “reference” glass window is used to generate a template polarization pattern, differences between this template polarization pattern and a measured polarization pattern from another glass window may cause the detection of defects if the threshold is set too low, but if the threshold is set higher, then defects may go undetected. Some embodiments use an adaptive threshold and/or a threshold that is set based on physics-based priors. For example, if the surface is curved, then regions with high curvature are more likely to have stronger polarization signals. Therefore, in some embodiments, the threshold for this region is set differently than for a region that is estimated or expected to be flat. This adaptive thresholding can be very large (e.g., the threshold may differ by orders of magnitude between different surfaces), as the polarization strength can vary by two orders of magnitude between surfaces which appear mostly flat versus curved.

Accordingly, some aspects of embodiments of the present disclosure relate to an anomaly detection approach to detecting surface features in objects. For example, in some embodiments of the present disclosure, tensors in representation space are extracted from a large collection of known good reference samples. These reference tensors in representation space may differ from one another in accordance with natural variation (e.g., natural variations in their polarization patterns). Accordingly, one or more summary metrics can be computed on these reference tensors in representation space to cluster the various reference tensors, such as computing maxima and minima of DOLP, or characterizing the distribution of AOLP across different portions of the surface, or the smoothness of transitions in different levels of DOLP. The statistical distributions of these summary metrics of the set of known good objects may then be stored as a part of a stored model 810 for characterizing a surface.

In these embodiments of the present disclosure, based on this approach, the stored model 810 includes an anomaly detection model as a statistical model for commonly expected characteristics of a particular surface of the object that is loaded based on registration of the raw polarization frames 18 (or the computed tensors in representations spaces 50), similar summary metrics are computed from measurements are performed on the computed tensors 50 from the surface under inspection. If these summary metrics for the surface under inspection are within the distribution of metrics from the known good samples as represented in the anomaly detection model, then this particular portion of the surface may be marked as being clean or defect free. On the other hand, if one or more of these measurements is outside of the distribution of measurements (e.g., more than a threshold distance away from the distribution of known good samples, such as more than two standard deviations away from the mean) then the surface may be marked as containing a defect.

Surface Characteristic Detection Using Trained Convolutional Neural Networks

In some embodiments of the present disclosure, the stored models 810 include trained convolutional neural networks (CNNs) that are trained to detect one or more defects in the surfaces of the objects based on the supplied tensors in representation spaces. These CNNs may be trained based on labeled training data (e.g., data in which training tensors in the representation spaces are used to train the weights of connections it the neural network to compute outputs that label defective portions in accordance with labeled training data).

In some embodiments of the present disclosure, the models are implemented using one or more of: encoder-decoder neural networks, or U-net architectures for semantic segmentation of defects. A U-net enables multiscale information to be propagated. In some embodiments of the present disclosure, a CNN architecture for semantic segmentation and/or instance segmentation is trained using polarization training data (e.g., training data including polarization raw frames as training input and segmentation masks as labeled training output).

One embodiment of the present disclosure using deep instance segmentation is based on a modification of a Mask Region-based Convolutional Neural Network (Mask R-CNN) architecture to form a Polarized Mask R-CNN architecture. Mask R-CNN works by taking an input image x, which is an H×W×3 tensor of image intensity values (e.g., height by width by color intensity in red, green, and blue channels), and running it through a backbone network: C=B(x). The backbone network B(x) is responsible for extracting useful learned features from the input image and can be any standard CNN architecture such as AlexNet (see, e.g., Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.), VGG (see, e.g., Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014).), ResNet-101 (see, e.g., Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.), MobileNet (see, e.g., Howard, Andrew G., et al. “Mobilenets: Efficient convolutional neural networks for mobile vision applications.” arXiv preprint arXiv:1704.04861 (2017).), MobileNetV2 (see, e.g., Sandler, Mark, et al. “MobileNetV2: Inverted residuals and linear bottlenecks.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.), and MobileNetV3 (see, e.g., Howard, Andrew, et al. “Searching for MobileNetV3.” Proceedings of the IEEE International Conference on Computer Vision. 2019.)

The backbone network B(x) outputs a set of tensors, e.g., custom character ={₁, ₂, ₃, ₄, ₅}, where each tensor _irepresents a different resolution feature map. These feature maps are then combined in a feature pyramid network (FPN) (see, e.g., Tsung-Yi Lin, Piotr Doll'ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117-2125, 2017.), processed with a region proposal network (RPN) (see, e.g., Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91-99, 2015.), and finally passed through an output subnetwork (see, e.g., Ren et al. and He et al., above) to produce classes, bounding boxes, and pixel-wise segmentations. These are merged with non-maximum suppression for instance segmentation.

In some embodiments, a Mask R-CNN architecture is used as a component of a Polarized Mask R-CNN architecture that is configured to take several input tensors, including tensors in polarization representation spaces, and to compute multi-scale second tensors in second representation spaces. In some embodiments, the tensors in the different first representation spaces are referred to as being in different “modes,” and the tensors of each mode may be supplied to a separate Mask R-CNN backbone for each mode. Each of these backbones computes mode tensors at multiple scales or resolutions (e.g., corresponding to different scaled versions of the input first tensors), and the mode tensors computed at each scale for the different modes are fused to generate a fused tensor for each of the scales. The fused tensors or second tensors may then be supplied to a prediction module, which is trained to compute a prediction (e.g., identification of surface characteristics) based on the fused tensors or second tensors. A Polarized Mask R-CNN architecture is described in more detail in U.S. Provisional Patent Application No. 63/001,445, filed in the United States Patent and Trademark Office on Mar. 29, 2020 and in International Patent Application No. PCT/US20/48604, filed in the United States Patent and Trademark Office on Aug. 28, 2020, the entire disclosures of which are incorporated by reference herein.

While some embodiments of the present disclosure relate to surface characterization using a Polarized CNN architecture that includes a Mask R-CNN backbone, embodiments of the present disclosure are not limited thereto, and other backbones such as AlexNet, VGG, MobileNet, MobileNetV2, MobileNetV3, and the like may be modified in a similar manner in place of one or more (e.g., in place of all) of the Mask R-CNN backbones.

Accordingly, in some embodiments of the present disclosure, surface characterization results 20 are computed by supplying the first tensors, including tensors in polarization feature representation spaces, to a trained convolutional neural network (CNN), such as a Polarized Mask R-CNN architecture, to compute a segmentation map, where the segmentation map identifies locations or portions of the input images (e.g., the input polarization raw frames) that correspond to particular surface characteristics (e.g., surface defects such as cracks, dents, uneven paint, the presence of surface contaminants, and the like or surface features such as surface smoothness versus roughness, surface flatness versus curvature, and the like).

Surface Characteristic Detection Using Classifiers

In some embodiments of the present disclosure, rather than use a convolutional neural network to identify regions of the surface under inspection that contain various surface characteristics of interest (e.g., that contain defects), the model 810 includes a trained classifier that classifies the given input into one or more categories. For example, a trained classifier may compute a characterization output 20 that includes a vector having a length equal to the number of different possible surface characteristics that the classifier is trained to detect, where each value in the vector corresponds to a confidence that the input image depicts the corresponding surface characteristic.

A classifier may be trained to take input images of a fixed size, where the inputs may be computed by, for example, extracting first tensors in first representation spaces from the raw polarization frames and supplying the entire first tensors as input to the classifier or dividing the first tensors into fixed size blocks. In various embodiments of the present disclosure, the classifier may include, for example, a support vector machine, a deep neural network (e.g., a deep fully connected neural network), and the like.

Training Data for Training Statistical Models

Some aspects of embodiments of the present disclosure relate to preparing training data for training statistical models for detecting surface features. In some circumstances, manually labeled (e.g., human labeled) training data may be available, such as in the form of manually capturing polarization raw frames of the surface of an object using a polarization camera and labeling regions of the images as containing surface characteristics of interest (e.g., borders between different types of materials, locations of defects such as dents and cracks, or surface irregularities such as rough portions of a surface that is expected to be smooth). These manually labeled training data may be used as part of a training set for training a statistical model such as an anomaly detector or a convolutional neural network as described above.

While manually labeled training data is generally considered to be good training data, there may be circumstances in which this manually labeled data may be insufficiently large to train a good statistical model. As such, some aspects of embodiments of the present disclosure further relate to augmenting a training data set, which may include synthesizing additional training data.

In some embodiments of the present disclosure, computer graphics techniques are used to synthesize training object data with and without the surface characteristics of interest. For example, when training a detector to detect surface defects, polarization raw frames of defect-free surfaces may be combined with polarization raw frames depicting defects such as cracks, chips, burrs, uneven paint, and the like. These separate images may be combined using computer graphics techniques (e.g., image editing tools to programmatically clone or composite the polarization raw frame images of the defects onto the polarization raw frames of defect-free surfaces to simulate or synthesize polarization raw frames of surfaces containing defects). The composited defects may be placed on physically reasonable locations of the clean surfaces (e.g., an image of a dent in a door panel is composited into images of portions of door panels that can be dented and not placed in physically unrealistic areas such as on glass windows, likewise, a chip in a glass surface may be composited into glass surfaces but not onto images of plastic trim).

As another example, in some embodiments of the present disclosure, a generative adversarial network (GAN) is trained to generate synthesized data, where a generative network is trained to synthesize polarization raw frames of surfaces depicting defects and a judging network is trained to determine whether its inputs are genuine polarization raw frames or synthesized (e.g., by the generative network).

In some embodiments of the present disclosure, a technique known as “domain randomization” is used to add “random” image-based perturbations to the simulated or synthesized training data to make the synthesized training data more closely resemble real-world data. For example, in some embodiments of the present disclosure, rotation augmentation is applied to the training data to augment the training data with rotated versions of the various features. This may be particularly beneficial to the accuracy of detection of defects that have extreme aspect ratios (e.g., scratches) that are not well-represented in natural images.

In various embodiments of the present disclosure, a statistical model is trained using the training data based on corresponding techniques. For example, in embodiments using an anomaly detection approach, various statistics are computed on the sets of good data, such as the mean and the variance of the good data points to determine threshold distances (e.g., two standard deviations) for determining whether a given sample is acceptable or is anomalous (e.g., defective). In embodiments using a neural network such as a convolutional neural network (e.g., a Polarization Mask R-CNN), the training process may include updating the weights of connections between neurons of various layers of the neural network in accordance with a backpropagation algorithm and the use of gradient descent to iteratively adjust the weights to minimize an error (or loss) between the output of the neural network and the labeled training data.

As such, aspects of embodiments of the present disclosure provide systems and methods for automatic characterization of surfaces, such as for the automated inspection of manufactured parts as they roll off the assembly line. These automation processes enable cost savings for manufacturers, not only through automation and consequent reduction of manual labor in inspection, but also through robust and accurate handling of anomalies in the products themselves (e.g., automatically removing defective products from a manufacturing stream).

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, and equivalents thereof.

Claims

1. A computer-implemented method for surface modeling, the method comprising: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames being captured at different polarizations by a polarization camera comprising a polarizing filter;extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; anddetecting a surface characteristic of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces, comprising: loading a stored model corresponding to a location of the surface of the physical object, the stored model comprising one or more reference tensors in the one or more polarization representation spaces; andcomputing the surface characteristic in accordance with the stored model and the one or more first tensors in the one or more polarization representation spaces comprising computing a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.
2. The computer-implemented method of claim 1, wherein the one or more first tensors in the one or more polarization representation spaces comprise: a degree of linear polarization (DOLP) image in a DOLP representation space; andan angle of linear polarization (AOLP) image in an AOLP representation space.
3. The computer-implemented method of claim 1, wherein the one or more first tensors further comprise one or more non-polarization tensors in one or more non-polarization representation spaces, and wherein the one or more non-polarization tensors comprise one or more intensity images in intensity representation space.
4. The computer-implemented method of claim 3, wherein the one or more intensity images comprise: a first color intensity image;a second color intensity image; anda third color intensity image.
5. The computer-implemented method of claim 1, wherein the surface characteristic comprises a detection of a defect in the surface of the physical object.
6. The computer-implemented method of claim 1, wherein the difference is computed using a Fresnel distance.
7. The computer-implemented method of claim 1, wherein the stored model comprises a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.
8. The computer-implemented method of claim 7, wherein the trained statistical model comprises an anomaly detection model.
9. The computer-implemented method of claim 7, wherein the trained statistical model comprises a convolutional neural network trained to detect defects in the surface of the physical object.
10. The computer-implemented method of claim 7, wherein the trained statistical model comprises a trained classifier trained to detect defects.
11. A system for surface modeling, the system comprising: a polarization camera comprising a polarizing filter, the polarization camera being configured to capture polarization raw frames at different polarizations; anda processing system comprising a processor and memory storing instructions that, when executed by the processor, cause the processor to: receive one or more polarization raw frames of a surface of a physical object, the polarization raw frames corresponding to different polarizations of light;extract one or more first tensors in one or more polarization representation spaces from the polarization raw frames; anddetect a surface characteristic of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces, the instructions further comprising instructions that, when executed by the processor, cause the processor to detect the stored characteristic by: loading a stored model corresponding to a location of the surface of the physical object, the stored model comprising one or more reference tensors in the one or more polarization representation spaces; andcomputing the surface characteristic in accordance with the stored model and the one or more first tensors in the one or more polarization representation spaces by computing a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.
12. The system of claim 11, wherein the one or more first tensors in the one or more polarization representation spaces comprise: a degree of linear polarization (DOLP) image in a DOLP representation space; andan angle of linear polarization (AOLP) image in an AOLP representation space.
13. The system claim 11, wherein the one or more first tensors further comprise one or more non-polarization tensors in one or more non-polarization representation spaces, and wherein the one or more non-polarization tensors comprise one or more intensity images in intensity representation space.
14. The system of claim 13, wherein the one or more intensity images comprise: a first color intensity image;a second color intensity image; anda third color intensity image.
15. The system of claim 11, wherein the surface characteristic comprises a detection of a defect in the surface of the physical object.
16. The system of claim 11, wherein the difference is computed using a Fresnel distance.
17. The system of claim 11, wherein the stored model comprises a trained statistical model configured to compute a prediction of the surface characteristic based on the one or more first tensors in the one or more polarization representation spaces.
18. The system of claim 17, wherein the trained statistical model comprises an anomaly detection model.
19. The system of claim 17, wherein the trained statistical model comprises a convolutional neural network trained to detect defects in the surface of the physical object.
20. The system of claim 17, wherein the trained statistical model comprises a trained classifier trained to detect defects.
21. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving one or more polarization raw frames of a surface of a physical object, the polarization raw frames being captured at different polarizations by a polarization camera comprising a polarizing filter;extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; anddetecting a surface characteristic of the surface of the physical object based on the one or more first tensors in the one or more polarization representation spaces, comprising: loading a stored model corresponding to a location of the surface of the physical object, the stored model comprising one or more reference tensors in the one or more polarization representation spaces; andcomputing the surface characteristic in accordance with the stored model and the one or more first tensors in the one or more polarization representation spaces comprising computing a difference between the one or more reference tensors and the one or more first tensors in the one or more polarization representation spaces.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No. 17/266,054 filed Feb. 4, 2021, which is a U.S. National Phase Patent Application of International Application Number PCT/US2020/051243, filed on Sep. 17, 2020, which claims priority to and the benefit of U.S. Provisional Patent Application No. 62/901,731, filed in the United States Patent and Trademark Office on Sep. 17, 2019 and which claims priority to and the benefit of U.S. Provisional Patent Application No. 63/001,445, filed in the United States Patent and Trademark Office on Mar. 29, 2020, the entire disclosures of each of which is incorporated by reference herein.

US Referenced Citations (1306)

Number	Name	Date	Kind
4124798	Thompson	Nov 1978	A
4198646	Alexander et al.	Apr 1980	A
4323925	Abell et al.	Apr 1982	A
4460449	Montalbano	Jul 1984	A
4467365	Murayama et al.	Aug 1984	A
4652909	Glenn	Mar 1987	A
4888645	Mitchell et al.	Dec 1989	A
4899060	Lischke	Feb 1990	A
4962425	Rea	Oct 1990	A
5005083	Grage et al.	Apr 1991	A
5070414	Tsutsumi	Dec 1991	A
5144448	Hornbaker et al.	Sep 1992	A
5157499	Oguma et al.	Oct 1992	A
5325449	Burt et al.	Jun 1994	A
5327125	Iwase et al.	Jul 1994	A
5463464	Ladewski	Oct 1995	A
5475422	Suzuki et al.	Dec 1995	A
5488674	Burt et al.	Jan 1996	A
5517236	Sergeant et al.	May 1996	A
5629524	Stettner et al.	May 1997	A
5638461	Fridge	Jun 1997	A
5675377	Gibas et al.	Oct 1997	A
5703961	Rogina et al.	Dec 1997	A
5710875	Hsu et al.	Jan 1998	A
5757425	Barton et al.	May 1998	A
5793900	Nourbakhsh et al.	Aug 1998	A
5801919	Griencewic	Sep 1998	A
5808350	Jack et al.	Sep 1998	A
5832312	Rieger et al.	Nov 1998	A
5833507	Woodgate et al.	Nov 1998	A
5867584	Hu et al.	Feb 1999	A
5880691	Fossum et al.	Mar 1999	A
5911008	Niikura et al.	Jun 1999	A
5933190	Dierickx et al.	Aug 1999	A
5963664	Kumar et al.	Oct 1999	A
5973844	Burger	Oct 1999	A
6002743	Telymonde	Dec 1999	A
6005607	Uomori et al.	Dec 1999	A
6034690	Gallery et al.	Mar 2000	A
6069351	Mack	May 2000	A
6069365	Chow et al.	May 2000	A
6084979	Kanade et al.	Jul 2000	A
6095989	Hay et al.	Aug 2000	A
6097394	Levoy et al.	Aug 2000	A
6124974	Burger	Sep 2000	A
6130786	Osawa et al.	Oct 2000	A
6137100	Fossum et al.	Oct 2000	A
6137535	Meyers	Oct 2000	A
6141048	Meyers	Oct 2000	A
6160909	Melen	Dec 2000	A
6163414	Kikuchi et al.	Dec 2000	A
6172352	Liu	Jan 2001	B1
6175379	Uomori et al.	Jan 2001	B1
6185529	Chen et al.	Feb 2001	B1
6198852	Anandan et al.	Mar 2001	B1
6205241	Melen	Mar 2001	B1
6239909	Hayashi et al.	May 2001	B1
6292713	Jouppi et al.	Sep 2001	B1
6340994	Margulis et al.	Jan 2002	B1
6358862	Ireland et al.	Mar 2002	B1
6373518	Sogawa	Apr 2002	B1
6419638	Hay et al.	Jul 2002	B1
6443579	Myers	Sep 2002	B1
6445815	Sato	Sep 2002	B1
6476805	Shume et al.	Nov 2002	B1
6477260	Shimomura	Nov 2002	B1
6502097	Chan et al.	Dec 2002	B1
6525302	Dowski, Jr. et al.	Feb 2003	B2
6546153	Hoydal	Apr 2003	B1
6552742	Seta	Apr 2003	B1
6563537	Kawamura et al.	May 2003	B1
6571466	Glenn et al.	Jun 2003	B1
6603513	Berezin	Aug 2003	B1
6611289	Yu et al.	Aug 2003	B1
6627896	Hashimoto et al.	Sep 2003	B1
6628330	Lin	Sep 2003	B1
6628845	Stone et al.	Sep 2003	B1
6635941	Suda	Oct 2003	B2
6639596	Shum et al.	Oct 2003	B1
6647142	Beardsley	Nov 2003	B1
6657218	Noda	Dec 2003	B2
6671399	Berestov	Dec 2003	B1
6674892	Melen	Jan 2004	B1
6750488	Driescher et al.	Jun 2004	B1
6750904	Lambert	Jun 2004	B1
6765617	Tangen et al.	Jul 2004	B1
6771833	Edgar	Aug 2004	B1
6774941	Boisvert et al.	Aug 2004	B1
6788338	Dinev et al.	Sep 2004	B1
6795253	Shinohara	Sep 2004	B2
6801653	Wu et al.	Oct 2004	B1
6819328	Moriwaki et al.	Nov 2004	B1
6819358	Kagle et al.	Nov 2004	B1
6833863	Clemens	Dec 2004	B1
6879735	Portniaguine et al.	Apr 2005	B1
6897454	Sasaki et al.	May 2005	B2
6903770	Kobayashi et al.	Jun 2005	B1
6909121	Nishikawa	Jun 2005	B2
6917702	Beardsley	Jul 2005	B2
6927922	George et al.	Aug 2005	B2
6958862	Joseph	Oct 2005	B1
6985175	Iwai et al.	Jan 2006	B2
7013318	Rosengard et al.	Mar 2006	B2
7015954	Foote et al.	Mar 2006	B1
7085409	Sawhney et al.	Aug 2006	B2
7161614	Yamashita et al.	Jan 2007	B1
7199348	Olsen et al.	Apr 2007	B2
7206449	Raskar et al.	Apr 2007	B2
7215364	Wachtel et al.	May 2007	B2
7235785	Hornback et al.	Jun 2007	B2
7245761	Swaminathan et al.	Jul 2007	B2
7262799	Suda	Aug 2007	B2
7292735	Blake et al.	Nov 2007	B2
7295697	Satoh	Nov 2007	B1
7333651	Kim et al.	Feb 2008	B1
7369165	Bosco et al.	May 2008	B2
7391572	Jacobowitz et al.	Jun 2008	B2
7408725	Sato	Aug 2008	B2
7425984	Chen et al.	Sep 2008	B2
7430312	Gu	Sep 2008	B2
7471765	Jaffray et al.	Dec 2008	B2
7496293	Shamir et al.	Feb 2009	B2
7564019	Olsen et al.	Jul 2009	B2
7599547	Sun et al.	Oct 2009	B2
7606484	Richards et al.	Oct 2009	B1
7620265	Wolff et al.	Nov 2009	B1
7633511	Shum et al.	Dec 2009	B2
7639435	Chiang	Dec 2009	B2
7639838	Nims	Dec 2009	B2
7646549	Zalevsky et al.	Jan 2010	B2
7657090	Omatsu et al.	Feb 2010	B2
7667824	Moran	Feb 2010	B1
7675080	Boettiger	Mar 2010	B2
7675681	Tomikawa et al.	Mar 2010	B2
7706634	Schmitt et al.	Apr 2010	B2
7723662	Levoy et al.	May 2010	B2
7738013	Galambos et al.	Jun 2010	B2
7741620	Doering et al.	Jun 2010	B2
7782364	Smith	Aug 2010	B2
7826153	Hong	Nov 2010	B2
7840067	Shen et al.	Nov 2010	B2
7912673	Hébert et al.	Mar 2011	B2
7924321	Nayar et al.	Apr 2011	B2
7956871	Fainstain et al.	Jun 2011	B2
7965314	Miller et al.	Jun 2011	B1
7973834	Yang	Jul 2011	B2
7986018	Rennie	Jul 2011	B2
7990447	Honda et al.	Aug 2011	B2
8000498	Shih et al.	Aug 2011	B2
8013904	Tan et al.	Sep 2011	B2
8027531	Wilburn et al.	Sep 2011	B2
8044994	Vetro et al.	Oct 2011	B2
8055466	Bryll	Nov 2011	B2
8077245	Adamo et al.	Dec 2011	B2
8089515	Chebil et al.	Jan 2012	B2
8098297	Crisan et al.	Jan 2012	B2
8098304	Pinto et al.	Jan 2012	B2
8106949	Tan et al.	Jan 2012	B2
8111910	Tanaka	Feb 2012	B2
8126279	Marcellin et al.	Feb 2012	B2
8130120	Kawabata et al.	Mar 2012	B2
8131097	Lelescu et al.	Mar 2012	B2
8149323	Li et al.	Apr 2012	B2
8164629	Zhang	Apr 2012	B1
8169486	Corcoran et al.	May 2012	B2
8180145	Wu et al.	May 2012	B2
8189065	Georgiev et al.	May 2012	B2
8189089	Georgiev et al.	May 2012	B1
8194296	Compton et al.	Jun 2012	B2
8212914	Chiu	Jul 2012	B2
8213711	Tam	Jul 2012	B2
8231814	Duparre	Jul 2012	B2
8242426	Ward et al.	Aug 2012	B2
8244027	Takahashi	Aug 2012	B2
8244058	Intwala et al.	Aug 2012	B1
8254668	Mashitani et al.	Aug 2012	B2
8279325	Pitts et al.	Oct 2012	B2
8280194	Wong et al.	Oct 2012	B2
8284240	Saint-Pierre et al.	Oct 2012	B2
8289409	Chang	Oct 2012	B2
8289440	Pitts et al.	Oct 2012	B2
8290358	Georgiev	Oct 2012	B1
8294099	Blackwell, Jr.	Oct 2012	B2
8294754	Jung et al.	Oct 2012	B2
8300085	Yang et al.	Oct 2012	B2
8305456	McMahon	Nov 2012	B1
8315476	Georgiev et al.	Nov 2012	B1
8345144	Georgiev et al.	Jan 2013	B1
8350957	Schechner et al.	Jan 2013	B2
8360574	Ishak et al.	Jan 2013	B2
8400555	Georgiev et al.	Mar 2013	B1
8406562	Bassi et al.	Mar 2013	B2
8411146	Twede	Apr 2013	B2
8416282	Lablans	Apr 2013	B2
8446492	Nakano et al.	May 2013	B2
8456517	Spektor et al.	Jun 2013	B2
8471895	Banks	Jun 2013	B2
8493496	Freedman et al.	Jul 2013	B2
8514291	Chang	Aug 2013	B2
8514491	Duparre	Aug 2013	B2
8541730	Inuiya	Sep 2013	B2
8542933	Venkataraman et al.	Sep 2013	B2
8553093	Wong et al.	Oct 2013	B2
8558929	Tredwell	Oct 2013	B2
8559705	Ng	Oct 2013	B2
8559756	Georgiev et al.	Oct 2013	B2
8565547	Strandemar	Oct 2013	B2
8576302	Yoshikawa	Nov 2013	B2
8577183	Robinson	Nov 2013	B2
8581995	Lin et al.	Nov 2013	B2
8619082	Ciurea et al.	Dec 2013	B1
8648918	Kauker et al.	Feb 2014	B2
8648919	Mantzel et al.	Feb 2014	B2
8655052	Spooner et al.	Feb 2014	B2
8682107	Yoon et al.	Mar 2014	B2
8687087	Pertsel et al.	Apr 2014	B2
8692893	McMahon	Apr 2014	B2
8754941	Sarwari et al.	Jun 2014	B1
8773536	Zhang	Jul 2014	B1
8780113	Ciurea et al.	Jul 2014	B1
8787691	Takahashi et al.	Jul 2014	B2
8792710	Keselman	Jul 2014	B2
8804255	Duparre	Aug 2014	B2
8823813	Mantzel et al.	Sep 2014	B2
8830375	Ludwig	Sep 2014	B2
8831367	Venkataraman et al.	Sep 2014	B2
8831377	Pitts et al.	Sep 2014	B2
8836793	Kriesel et al.	Sep 2014	B1
8842201	Tajiri	Sep 2014	B2
8854433	Rafii	Oct 2014	B1
8854462	Herbin et al.	Oct 2014	B2
8861089	Duparre	Oct 2014	B2
8866912	Mullis	Oct 2014	B2
8866920	Venkataraman et al.	Oct 2014	B2
8866951	Keelan	Oct 2014	B2
8878950	Lelescu et al.	Nov 2014	B2
8885059	Venkataraman et al.	Nov 2014	B1
8885922	Ito et al.	Nov 2014	B2
8896594	Xiong et al.	Nov 2014	B2
8896719	Venkataraman et al.	Nov 2014	B1
8902321	Venkataraman et al.	Dec 2014	B2
8928793	McMahon	Jan 2015	B2
8977038	Tian et al.	Mar 2015	B2
9001226	Ng et al.	Apr 2015	B1
9019426	Han et al.	Apr 2015	B2
9025894	Venkataraman et al.	May 2015	B2
9025895	Venkataraman et al.	May 2015	B2
9030528	Pesach et al.	May 2015	B2
9031335	Venkataraman et al.	May 2015	B2
9031342	Venkataraman	May 2015	B2
9031343	Venkataraman	May 2015	B2
9036928	Venkataraman	May 2015	B2
9036931	Venkataraman et al.	May 2015	B2
9041823	Venkataraman et al.	May 2015	B2
9041824	Lelescu et al.	May 2015	B2
9041829	Venkataraman et al.	May 2015	B2
9042667	Venkataraman et al.	May 2015	B2
9047684	Lelescu et al.	Jun 2015	B2
9049367	Venkataraman et al.	Jun 2015	B2
9055233	Venkataraman et al.	Jun 2015	B2
9060120	Venkataraman et al.	Jun 2015	B2
9060124	Venkataraman et al.	Jun 2015	B2
9077893	Venkataraman et al.	Jul 2015	B2
9094661	Venkataraman et al.	Jul 2015	B2
9100586	McMahon et al.	Aug 2015	B2
9100635	Duparre et al.	Aug 2015	B2
9123117	Ciurea et al.	Sep 2015	B2
9123118	Ciurea et al.	Sep 2015	B2
9124815	Venkataraman et al.	Sep 2015	B2
9124831	Mullis	Sep 2015	B2
9124864	Mullis	Sep 2015	B2
9128228	Duparre	Sep 2015	B2
9129183	Venkataraman et al.	Sep 2015	B2
9129377	Ciurea et al.	Sep 2015	B2
9143711	McMahon	Sep 2015	B2
9147254	Florian et al.	Sep 2015	B2
9185276	Rodda et al.	Nov 2015	B2
9188765	Venkataraman et al.	Nov 2015	B2
9191580	Venkataraman et al.	Nov 2015	B2
9197821	McMahon	Nov 2015	B2
9210392	Nisenzon et al.	Dec 2015	B2
9214013	Venkataraman et al.	Dec 2015	B2
9235898	Venkataraman et al.	Jan 2016	B2
9235900	Ciurea et al.	Jan 2016	B2
9240049	Ciurea et al.	Jan 2016	B2
9247117	Jacques	Jan 2016	B2
9253380	Venkataraman et al.	Feb 2016	B2
9253397	Lee et al.	Feb 2016	B2
9256974	Hines	Feb 2016	B1
9264592	Rodda et al.	Feb 2016	B2
9264610	Duparre	Feb 2016	B2
9361662	Lelescu et al.	Jun 2016	B2
9374512	Venkataraman et al.	Jun 2016	B2
9412206	McMahon et al.	Aug 2016	B2
9413953	Maeda	Aug 2016	B2
9426343	Rodda et al.	Aug 2016	B2
9426361	Venkataraman et al.	Aug 2016	B2
9438888	Venkataraman et al.	Sep 2016	B2
9445003	Lelescu et al.	Sep 2016	B1
9456134	Venkataraman et al.	Sep 2016	B2
9456196	Kim et al.	Sep 2016	B2
9462164	Venkataraman et al.	Oct 2016	B2
9485496	Venkataraman et al.	Nov 2016	B2
9497370	Venkataraman et al.	Nov 2016	B2
9497429	Mullis et al.	Nov 2016	B2
9516222	Duparre et al.	Dec 2016	B2
9519972	Venkataraman et al.	Dec 2016	B2
9521319	Rodda et al.	Dec 2016	B2
9521416	McMahon et al.	Dec 2016	B1
9536166	Venkataraman et al.	Jan 2017	B2
9576369	Venkataraman et al.	Feb 2017	B2
9578237	Duparre et al.	Feb 2017	B2
9578259	Molina	Feb 2017	B2
9602805	Venkataraman et al.	Mar 2017	B2
9633442	Venkataraman et al.	Apr 2017	B2
9635274	Lin et al.	Apr 2017	B2
9638883	Duparre	May 2017	B1
9661310	Deng et al.	May 2017	B2
9706132	Nisenzon et al.	Jul 2017	B2
9712759	Venkataraman et al.	Jul 2017	B2
9729865	Kuo et al.	Aug 2017	B1
9733486	Lelescu et al.	Aug 2017	B2
9741118	Mullis	Aug 2017	B2
9743051	Venkataraman et al.	Aug 2017	B2
9749547	Venkataraman et al.	Aug 2017	B2
9749568	McMahon	Aug 2017	B2
9754422	McMahon et al.	Sep 2017	B2
9766380	Duparre et al.	Sep 2017	B2
9769365	Jannard	Sep 2017	B1
9774789	Ciurea et al.	Sep 2017	B2
9774831	Venkataraman et al.	Sep 2017	B2
9787911	McMahon et al.	Oct 2017	B2
9794476	Nayar et al.	Oct 2017	B2
9800856	Venkataraman et al.	Oct 2017	B2
9800859	Venkataraman et al.	Oct 2017	B2
9807382	Duparre et al.	Oct 2017	B2
9811753	Venkataraman et al.	Nov 2017	B2
9813616	Lelescu et al.	Nov 2017	B2
9813617	Venkataraman et al.	Nov 2017	B2
9826212	Newton et al.	Nov 2017	B2
9858673	Ciurea et al.	Jan 2018	B2
9864921	Venkataraman et al.	Jan 2018	B2
9866739	McMahon	Jan 2018	B2
9888194	Duparre	Feb 2018	B2
9892522	Smirnov et al.	Feb 2018	B2
9898856	Yang et al.	Feb 2018	B2
9917998	Venkataraman et al.	Mar 2018	B2
9924092	Rodda et al.	Mar 2018	B2
9936148	McMahon	Apr 2018	B2
9942474	Venkataraman et al.	Apr 2018	B2
9955070	Lelescu et al.	Apr 2018	B2
9986224	Mullis	May 2018	B2
10009538	Venkataraman et al.	Jun 2018	B2
10019816	Venkataraman et al.	Jul 2018	B2
10027901	Venkataraman et al.	Jul 2018	B2
10089740	Srikanth et al.	Oct 2018	B2
10091405	Molina	Oct 2018	B2
10119808	Venkataraman et al.	Nov 2018	B2
10122993	Venkataraman et al.	Nov 2018	B2
10127682	Mullis	Nov 2018	B2
10142560	Venkataraman et al.	Nov 2018	B2
10182216	Mullis et al.	Jan 2019	B2
10218889	McMahan	Feb 2019	B2
10225543	Mullis	Mar 2019	B2
10250871	Ciurea et al.	Apr 2019	B2
10261219	Duparre et al.	Apr 2019	B2
10275676	Venkataraman et al.	Apr 2019	B2
10306120	Duparre	May 2019	B2
10311649	McMohan et al.	Jun 2019	B2
10334241	Duparre et al.	Jun 2019	B2
10366472	Lelescu et al.	Jul 2019	B2
10375302	Nayar et al.	Aug 2019	B2
10375319	Venkataraman et al.	Aug 2019	B2
10380752	Ciurea et al.	Aug 2019	B2
10390005	Nisenzon et al.	Aug 2019	B2
10412314	McMahon et al.	Sep 2019	B2
10430682	Venkataraman et al.	Oct 2019	B2
10455168	McMahon	Oct 2019	B2
10455218	Venkataraman et al.	Oct 2019	B2
10462362	Lelescu et al.	Oct 2019	B2
10482618	Jain et al.	Nov 2019	B2
10540806	Yang et al.	Jan 2020	B2
10542208	Lelescu et al.	Jan 2020	B2
10547772	Molina	Jan 2020	B2
10560684	Mullis	Feb 2020	B2
10574905	Srikanth et al.	Feb 2020	B2
10638099	Mullis et al.	Apr 2020	B2
10643383	Venkataraman	May 2020	B2
10659751	Briggs	May 2020	B1
10674138	Venkataraman et al.	Jun 2020	B2
10694114	Venkataraman et al.	Jun 2020	B2
10708492	Venkataraman et al.	Jul 2020	B2
10735635	Duparre	Aug 2020	B2
10742861	McMahon	Aug 2020	B2
10767981	Venkataraman et al.	Sep 2020	B2
10805589	Venkataraman et al.	Oct 2020	B2
10818026	Jain et al.	Oct 2020	B2
10839485	Lelescu et al.	Nov 2020	B2
10909707	Ciurea et al.	Feb 2021	B2
10944961	Ciurea et al.	Mar 2021	B2
10958892	Mullis	Mar 2021	B2
10976239	Hart et al.	Apr 2021	B1
10984276	Venkataraman et al.	Apr 2021	B2
11022725	Duparre et al.	Jun 2021	B2
11024046	Venkataraman	Jun 2021	B2
20010005225	Clark et al.	Jun 2001	A1
20010019621	Hanna et al.	Sep 2001	A1
20010028038	Hamaguchi et al.	Oct 2001	A1
20010038387	Tomooka et al.	Nov 2001	A1
20020003669	Kedar et al.	Jan 2002	A1
20020012056	Trevino et al.	Jan 2002	A1
20020015536	Warren et al.	Feb 2002	A1
20020027608	Johnson et al.	Mar 2002	A1
20020028014	Ono	Mar 2002	A1
20020039438	Mori et al.	Apr 2002	A1
20020057845	Fossum et al.	May 2002	A1
20020061131	Sawhney et al.	May 2002	A1
20020063807	Margulis	May 2002	A1
20020075450	Aratani et al.	Jun 2002	A1
20020087403	Meyers et al.	Jul 2002	A1
20020089596	Yasuo	Jul 2002	A1
20020094027	Sato et al.	Jul 2002	A1
20020101528	Lee et al.	Aug 2002	A1
20020113867	Takigawa et al.	Aug 2002	A1
20020113888	Sonoda et al.	Aug 2002	A1
20020118113	Oku et al.	Aug 2002	A1
20020120634	Min et al.	Aug 2002	A1
20020122113	Foote	Sep 2002	A1
20020163054	Suda	Nov 2002	A1
20020167537	Trajkovic	Nov 2002	A1
20020171666	Endo et al.	Nov 2002	A1
20020177054	Saitoh et al.	Nov 2002	A1
20020190991	Efran et al.	Dec 2002	A1
20020195548	Dowski, Jr. et al.	Dec 2002	A1
20030025227	Daniell	Feb 2003	A1
20030026474	Yano	Feb 2003	A1
20030086079	Barth et al.	May 2003	A1
20030124763	Fan et al.	Jul 2003	A1
20030140347	Varsa	Jul 2003	A1
20030156189	Utsumi et al.	Aug 2003	A1
20030179418	Wengender et al.	Sep 2003	A1
20030188659	Merry et al.	Oct 2003	A1
20030190072	Adkins et al.	Oct 2003	A1
20030198377	Ng	Oct 2003	A1
20030211405	Venkataraman	Nov 2003	A1
20030231179	Suzuki	Dec 2003	A1
20040003409	Berstis	Jan 2004	A1
20040008271	Hagimori et al.	Jan 2004	A1
20040012689	Tinnerino et al.	Jan 2004	A1
20040027358	Nakao	Feb 2004	A1
20040047274	Amanai	Mar 2004	A1
20040050104	Ghosh et al.	Mar 2004	A1
20040056966	Schechner et al.	Mar 2004	A1
20040061787	Liu et al.	Apr 2004	A1
20040066454	Otani et al.	Apr 2004	A1
20040071367	Irani et al.	Apr 2004	A1
20040075654	Hsiao et al.	Apr 2004	A1
20040096119	Williams et al.	May 2004	A1
20040100570	Shizukuishi	May 2004	A1
20040105021	Hu	Jun 2004	A1
20040114807	Lelescu et al.	Jun 2004	A1
20040141659	Zhang	Jul 2004	A1
20040151401	Sawhney et al.	Aug 2004	A1
20040165090	Ning	Aug 2004	A1
20040169617	Yelton et al.	Sep 2004	A1
20040170340	Tipping et al.	Sep 2004	A1
20040174439	Upton	Sep 2004	A1
20040179008	Gordon et al.	Sep 2004	A1
20040179834	Szajewski et al.	Sep 2004	A1
20040196379	Chen et al.	Oct 2004	A1
20040207600	Zhang et al.	Oct 2004	A1
20040207836	Chhibber et al.	Oct 2004	A1
20040212734	Macinnis et al.	Oct 2004	A1
20040213449	Safaee-Rad et al.	Oct 2004	A1
20040218809	Blake et al.	Nov 2004	A1
20040234873	Venkataraman	Nov 2004	A1
20040239782	Equitz et al.	Dec 2004	A1
20040239885	Jaynes et al.	Dec 2004	A1
20040240052	Minefuji et al.	Dec 2004	A1
20040251509	Choi	Dec 2004	A1
20040264806	Herley	Dec 2004	A1
20050006477	Patel	Jan 2005	A1
20050007461	Chou et al.	Jan 2005	A1
20050009313	Suzuki et al.	Jan 2005	A1
20050010621	Pinto et al.	Jan 2005	A1
20050012035	Miller	Jan 2005	A1
20050036135	Earthman et al.	Feb 2005	A1
20050036778	DeMonte	Feb 2005	A1
20050047678	Jones et al.	Mar 2005	A1
20050048690	Yamamoto	Mar 2005	A1
20050068436	Fraenkel et al.	Mar 2005	A1
20050083531	Millerd et al.	Apr 2005	A1
20050084179	Hanna et al.	Apr 2005	A1
20050111705	Waupotitsch et al.	May 2005	A1
20050117015	Cutler	Jun 2005	A1
20050128509	Tokkonen et al.	Jun 2005	A1
20050128595	Shimizu	Jun 2005	A1
20050132098	Sonoda et al.	Jun 2005	A1
20050134698	Schroeder et al.	Jun 2005	A1
20050134699	Nagashima	Jun 2005	A1
20050134712	Gruhlke et al.	Jun 2005	A1
20050147277	Higaki et al.	Jul 2005	A1
20050151759	Gonzalez-Banos et al.	Jul 2005	A1
20050168924	Wu et al.	Aug 2005	A1
20050175257	Kuroki	Aug 2005	A1
20050185711	Pfister et al.	Aug 2005	A1
20050203380	Sauer et al.	Sep 2005	A1
20050205785	Hornback et al.	Sep 2005	A1
20050219264	Shum et al.	Oct 2005	A1
20050219363	Kohler et al.	Oct 2005	A1
20050224843	Boemler	Oct 2005	A1
20050225654	Feldman et al.	Oct 2005	A1
20050265633	Piacentino et al.	Dec 2005	A1
20050275946	Choo et al.	Dec 2005	A1
20050286612	Takanashi	Dec 2005	A1
20050286756	Hong et al.	Dec 2005	A1
20060002635	Nestares et al.	Jan 2006	A1
20060007331	Izumi et al.	Jan 2006	A1
20060013318	Webb et al.	Jan 2006	A1
20060018509	Miyoshi	Jan 2006	A1
20060023197	Joel	Feb 2006	A1
20060023314	Boettiger et al.	Feb 2006	A1
20060028476	Sobel et al.	Feb 2006	A1
20060029270	Berestov et al.	Feb 2006	A1
20060029271	Miyoshi et al.	Feb 2006	A1
20060033005	Jerdev et al.	Feb 2006	A1
20060034003	Zalevsky	Feb 2006	A1
20060034531	Poon et al.	Feb 2006	A1
20060035415	Wood	Feb 2006	A1
20060038891	Okutomi et al.	Feb 2006	A1
20060039611	Rother et al.	Feb 2006	A1
20060046204	Ono et al.	Mar 2006	A1
20060049930	Zruya et al.	Mar 2006	A1
20060050980	Kohashi et al.	Mar 2006	A1
20060054780	Garrood et al.	Mar 2006	A1
20060054782	Olsen et al.	Mar 2006	A1
20060055811	Frtiz et al.	Mar 2006	A1
20060069478	Iwama	Mar 2006	A1
20060072029	Miyatake et al.	Apr 2006	A1
20060087747	Ohzawa et al.	Apr 2006	A1
20060098888	Morishita	May 2006	A1
20060103754	Wenstrand et al.	May 2006	A1
20060119597	Oshino	Jun 2006	A1
20060125936	Gruhike et al.	Jun 2006	A1
20060138322	Costello et al.	Jun 2006	A1
20060139475	Esch et al.	Jun 2006	A1
20060152803	Provitola	Jul 2006	A1
20060153290	Watabe et al.	Jul 2006	A1
20060157640	Perlman et al.	Jul 2006	A1
20060159369	Young	Jul 2006	A1
20060176566	Boettiger et al.	Aug 2006	A1
20060187322	Janson, Jr. et al.	Aug 2006	A1
20060187338	May et al.	Aug 2006	A1
20060197937	Bamji et al.	Sep 2006	A1
20060203100	Ajito et al.	Sep 2006	A1
20060203113	Wada et al.	Sep 2006	A1
20060210146	Gu	Sep 2006	A1
20060210186	Berkner	Sep 2006	A1
20060214085	Olsen et al.	Sep 2006	A1
20060215879	Whitaker	Sep 2006	A1
20060215924	Steinberg et al.	Sep 2006	A1
20060221250	Rossbach et al.	Oct 2006	A1
20060239549	Kelly et al.	Oct 2006	A1
20060243889	Farnworth et al.	Nov 2006	A1
20060251410	Trutna	Nov 2006	A1
20060274174	Tewinkle	Dec 2006	A1
20060278948	Yamaguchi et al.	Dec 2006	A1
20060279648	Senba et al.	Dec 2006	A1
20060289772	Johnson et al.	Dec 2006	A1
20070002159	Olsen et al.	Jan 2007	A1
20070008575	Yu et al.	Jan 2007	A1
20070009150	Suwa	Jan 2007	A1
20070024614	Tam et al.	Feb 2007	A1
20070030356	Yea et al.	Feb 2007	A1
20070035707	Margulis	Feb 2007	A1
20070036427	Nakamura et al.	Feb 2007	A1
20070040828	Zalevsky et al.	Feb 2007	A1
20070040922	McKee et al.	Feb 2007	A1
20070041391	Lin et al.	Feb 2007	A1
20070052825	Cho	Mar 2007	A1
20070083114	Yang et al.	Apr 2007	A1
20070085917	Kobayashi	Apr 2007	A1
20070092245	Bazakos et al.	Apr 2007	A1
20070102622	Olsen et al.	May 2007	A1
20070116447	Ye	May 2007	A1
20070126898	Feldman et al.	Jun 2007	A1
20070127831	Venkataraman	Jun 2007	A1
20070139333	Sato et al.	Jun 2007	A1
20070140685	Wu	Jun 2007	A1
20070146503	Shiraki	Jun 2007	A1
20070146511	Kinoshita et al.	Jun 2007	A1
20070153335	Hosaka	Jul 2007	A1
20070158427	Zhu et al.	Jul 2007	A1
20070159541	Sparks et al.	Jul 2007	A1
20070160310	Tanida et al.	Jul 2007	A1
20070165931	Higaki	Jul 2007	A1
20070166447	U r-Rehman et al.	Jul 2007	A1
20070171290	Kroger	Jul 2007	A1
20070177004	Kolehmainen et al.	Aug 2007	A1
20070182843	Shimamura et al.	Aug 2007	A1
20070201859	Sarrat	Aug 2007	A1
20070206241	Smith et al.	Sep 2007	A1
20070211164	Olsen et al.	Sep 2007	A1
20070216765	Wong et al.	Sep 2007	A1
20070225600	Weibrecht et al.	Sep 2007	A1
20070228256	Mentzer et al.	Oct 2007	A1
20070236595	Pan et al.	Oct 2007	A1
20070242141	Ciurea	Oct 2007	A1
20070247517	Zhang et al.	Oct 2007	A1
20070257184	Olsen et al.	Nov 2007	A1
20070258006	Olsen et al.	Nov 2007	A1
20070258706	Raskar et al.	Nov 2007	A1
20070263113	Baek et al.	Nov 2007	A1
20070263114	Gurevich et al.	Nov 2007	A1
20070268374	Robinson	Nov 2007	A1
20070291995	Rivera	Dec 2007	A1
20070296721	Chang et al.	Dec 2007	A1
20070296832	Ota et al.	Dec 2007	A1
20070296835	Olsen et al.	Dec 2007	A1
20070296846	Barman et al.	Dec 2007	A1
20070296847	Chang et al.	Dec 2007	A1
20070297696	Hamza et al.	Dec 2007	A1
20080006859	Mionetto	Jan 2008	A1
20080019611	Larkin et al.	Jan 2008	A1
20080024683	Damera-Venkata et al.	Jan 2008	A1
20080025649	Liu et al.	Jan 2008	A1
20080030592	Border et al.	Feb 2008	A1
20080030597	Olsen et al.	Feb 2008	A1
20080043095	Vetro et al.	Feb 2008	A1
20080043096	Vetro et al.	Feb 2008	A1
20080044170	Yap et al.	Feb 2008	A1
20080054518	Ra et al.	Mar 2008	A1
20080056302	Erdal et al.	Mar 2008	A1
20080062164	Bassi et al.	Mar 2008	A1
20080079805	Takagi et al.	Apr 2008	A1
20080080028	Bakin et al.	Apr 2008	A1
20080084486	Enge et al.	Apr 2008	A1
20080088793	Sverdrup et al.	Apr 2008	A1
20080095523	Schilling-Benz et al.	Apr 2008	A1
20080099804	Venezia et al.	May 2008	A1
20080106620	Sawachi	May 2008	A1
20080112059	Choi et al.	May 2008	A1
20080112635	Kondo et al.	May 2008	A1
20080117289	Schowengerdt et al.	May 2008	A1
20080118241	TeKolste et al.	May 2008	A1
20080131019	Ng	Jun 2008	A1
20080131107	Ueno	Jun 2008	A1
20080151097	Chen et al.	Jun 2008	A1
20080152213	Medioni et al.	Jun 2008	A1
20080152215	Horie et al.	Jun 2008	A1
20080152296	Oh et al.	Jun 2008	A1
20080156991	Hu et al.	Jul 2008	A1
20080158259	Kempf et al.	Jul 2008	A1
20080158375	Kakkori et al.	Jul 2008	A1
20080158698	Chang et al.	Jul 2008	A1
20080165257	Boettiger	Jul 2008	A1
20080174670	Olsen et al.	Jul 2008	A1
20080187305	Raskar et al.	Aug 2008	A1
20080193026	Horie et al.	Aug 2008	A1
20080208506	Kuwata	Aug 2008	A1
20080211737	Kim et al.	Sep 2008	A1
20080218610	Chapman et al.	Sep 2008	A1
20080218611	Parulski et al.	Sep 2008	A1
20080218612	Border et al.	Sep 2008	A1
20080218613	Janson et al.	Sep 2008	A1
20080219654	Border et al.	Sep 2008	A1
20080239116	Smith	Oct 2008	A1
20080240598	Hasegawa	Oct 2008	A1
20080246866	Kinoshita et al.	Oct 2008	A1
20080247638	Tanida et al.	Oct 2008	A1
20080247653	Moussavi et al.	Oct 2008	A1
20080272416	Yun	Nov 2008	A1
20080273751	Yuan et al.	Nov 2008	A1
20080278591	Barna et al.	Nov 2008	A1
20080278610	Boettiger	Nov 2008	A1
20080284880	Numata	Nov 2008	A1
20080291295	Kato et al.	Nov 2008	A1
20080298674	Baker et al.	Dec 2008	A1
20080310501	Ward et al.	Dec 2008	A1
20090027543	Kanehiro	Jan 2009	A1
20090050946	Duparre et al.	Feb 2009	A1
20090052743	Techmer	Feb 2009	A1
20090060281	Tanida et al.	Mar 2009	A1
20090066693	Carson	Mar 2009	A1
20090079862	Subbotin	Mar 2009	A1
20090086074	Li et al.	Apr 2009	A1
20090091645	Trimeche et al.	Apr 2009	A1
20090091806	Inuiya	Apr 2009	A1
20090092363	Daum et al.	Apr 2009	A1
20090096050	Park	Apr 2009	A1
20090102956	Georgiev	Apr 2009	A1
20090103792	Rahn et al.	Apr 2009	A1
20090109306	Shan et al.	Apr 2009	A1
20090127430	Hirasawa et al.	May 2009	A1
20090128644	Camp, Jr. et al.	May 2009	A1
20090128833	Yahav	May 2009	A1
20090129667	Ho et al.	May 2009	A1
20090140131	Utagawa	Jun 2009	A1
20090141933	Wagg	Jun 2009	A1
20090147919	Goto et al.	Jun 2009	A1
20090152664	Klem et al.	Jun 2009	A1
20090167922	Perlman et al.	Jul 2009	A1
20090167923	Safaee-Rad et al.	Jul 2009	A1
20090167934	Gupta	Jul 2009	A1
20090175349	Ye et al.	Jul 2009	A1
20090179142	Duparre et al.	Jul 2009	A1
20090180021	Kikuchi et al.	Jul 2009	A1
20090200622	Tai et al.	Aug 2009	A1
20090201371	Matsuda et al.	Aug 2009	A1
20090207235	Francini et al.	Aug 2009	A1
20090219435	Yuan	Sep 2009	A1
20090225203	Tanida et al.	Sep 2009	A1
20090237520	Kaneko et al.	Sep 2009	A1
20090237662	Chang et al.	Sep 2009	A1
20090245573	Saptharishi et al.	Oct 2009	A1
20090245637	Barman et al.	Oct 2009	A1
20090256947	Ciurea et al.	Oct 2009	A1
20090263017	Tanbakuchi	Oct 2009	A1
20090268192	Koenck et al.	Oct 2009	A1
20090268970	Babacan et al.	Oct 2009	A1
20090268983	Stone et al.	Oct 2009	A1
20090273663	Yoshida	Nov 2009	A1
20090274387	Jin	Nov 2009	A1
20090279800	Uetani et al.	Nov 2009	A1
20090284651	Srinivasan	Nov 2009	A1
20090290811	Imai	Nov 2009	A1
20090295933	Schechner et al.	Dec 2009	A1
20090297056	Lelescu et al.	Dec 2009	A1
20090302205	Olsen et al.	Dec 2009	A9
20090317061	Jung et al.	Dec 2009	A1
20090322876	Lee et al.	Dec 2009	A1
20090323195	Hembree et al.	Dec 2009	A1
20090323206	Oliver et al.	Dec 2009	A1
20090324118	Maslov et al.	Dec 2009	A1
20100002126	Wenstrand et al.	Jan 2010	A1
20100002313	Duparre et al.	Jan 2010	A1
20100002314	Duparre	Jan 2010	A1
20100007714	Kim et al.	Jan 2010	A1
20100013927	Nixon	Jan 2010	A1
20100044815	Chang	Feb 2010	A1
20100045809	Packard	Feb 2010	A1
20100053342	Hwang et al.	Mar 2010	A1
20100053347	Agarwala et al.	Mar 2010	A1
20100053415	Yun	Mar 2010	A1
20100053600	Tanida et al.	Mar 2010	A1
20100060746	Olsen et al.	Mar 2010	A9
20100073463	Momonoi et al.	Mar 2010	A1
20100074532	Gordon et al.	Mar 2010	A1
20100085351	Deb et al.	Apr 2010	A1
20100085425	Tan	Apr 2010	A1
20100086227	Sun et al.	Apr 2010	A1
20100091389	Henriksen et al.	Apr 2010	A1
20100097444	Lablans	Apr 2010	A1
20100097491	Farina et al.	Apr 2010	A1
20100103175	Okutomi et al.	Apr 2010	A1
20100103259	Tanida et al.	Apr 2010	A1
20100103308	Butterfield et al.	Apr 2010	A1
20100111444	Coffman	May 2010	A1
20100118127	Nam et al.	May 2010	A1
20100128145	Pitts et al.	May 2010	A1
20100129048	Pitts et al.	May 2010	A1
20100133230	Henriksen et al.	Jun 2010	A1
20100133418	Sargent et al.	Jun 2010	A1
20100141802	Knight et al.	Jun 2010	A1
20100142828	Chang et al.	Jun 2010	A1
20100142839	Lakus-Becker	Jun 2010	A1
20100157073	Kondo et al.	Jun 2010	A1
20100165152	Lim	Jul 2010	A1
20100166410	Chang	Jul 2010	A1
20100171866	Brady et al.	Jul 2010	A1
20100177411	Hegde et al.	Jul 2010	A1
20100182406	Benitez	Jul 2010	A1
20100194860	Mentz et al.	Aug 2010	A1
20100194901	van Hoorebeke et al.	Aug 2010	A1
20100195716	Klein Gunnewiek et al.	Aug 2010	A1
20100201809	Oyama et al.	Aug 2010	A1
20100201834	Maruyama et al.	Aug 2010	A1
20100202054	Niederer	Aug 2010	A1
20100202683	Robinson	Aug 2010	A1
20100208100	Olsen et al.	Aug 2010	A9
20100214423	Ogawa	Aug 2010	A1
20100220212	Perlman et al.	Sep 2010	A1
20100223237	Mishra et al.	Sep 2010	A1
20100225740	Jung et al.	Sep 2010	A1
20100231285	Boomer et al.	Sep 2010	A1
20100238327	Griffith et al.	Sep 2010	A1
20100244165	Lake et al.	Sep 2010	A1
20100245684	Xiao et al.	Sep 2010	A1
20100254627	Panahpour Tehrani et al.	Oct 2010	A1
20100259610	Petersen	Oct 2010	A1
20100265346	Iizuka	Oct 2010	A1
20100265381	Yamamoto et al.	Oct 2010	A1
20100265385	Knight et al.	Oct 2010	A1
20100277629	Tanaka	Nov 2010	A1
20100281070	Chan et al.	Nov 2010	A1
20100289941	Ito et al.	Nov 2010	A1
20100290483	Park et al.	Nov 2010	A1
20100296724	Chang et al.	Nov 2010	A1
20100302423	Adams, Jr. et al.	Dec 2010	A1
20100309292	Ho et al.	Dec 2010	A1
20100309368	Choi et al.	Dec 2010	A1
20100321595	Chiu	Dec 2010	A1
20100321640	Yeh et al.	Dec 2010	A1
20100329528	Hajnal	Dec 2010	A1
20100329556	Mitarai et al.	Dec 2010	A1
20100329582	Albu et al.	Dec 2010	A1
20110001037	Tewinkle	Jan 2011	A1
20110013006	Uzenbajakava et al.	Jan 2011	A1
20110018973	Takayama	Jan 2011	A1
20110019048	Raynor et al.	Jan 2011	A1
20110019243	Constant, Jr. et al.	Jan 2011	A1
20110031381	Tay et al.	Feb 2011	A1
20110032341	Ignatov et al.	Feb 2011	A1
20110032370	Ludwig	Feb 2011	A1
20110033129	Robinson	Feb 2011	A1
20110038536	Gong	Feb 2011	A1
20110043604	Peleg et al.	Feb 2011	A1
20110043613	Rohaly et al.	Feb 2011	A1
20110043661	Podoleanu	Feb 2011	A1
20110043665	Ogasahara	Feb 2011	A1
20110043668	McKinnon et al.	Feb 2011	A1
20110044502	Liu et al.	Feb 2011	A1
20110051255	Lee et al.	Mar 2011	A1
20110055729	Mason et al.	Mar 2011	A1
20110064327	Dagher et al.	Mar 2011	A1
20110069189	Venkataraman et al.	Mar 2011	A1
20110080487	Venkataraman et al.	Apr 2011	A1
20110084893	Lee et al.	Apr 2011	A1
20110085028	Samadani et al.	Apr 2011	A1
20110090217	Mashitani et al.	Apr 2011	A1
20110102553	Corcoran et al.	May 2011	A1
20110108708	Olsen et al.	May 2011	A1
20110115886	Nguyen et al.	May 2011	A1
20110121421	Charbon et al.	May 2011	A1
20110122308	Duparre	May 2011	A1
20110128393	Tavi et al.	Jun 2011	A1
20110128412	Milnes et al.	Jun 2011	A1
20110129165	Lim et al.	Jun 2011	A1
20110141309	Nagashima et al.	Jun 2011	A1
20110142138	Tian et al.	Jun 2011	A1
20110149408	Hahgholt et al.	Jun 2011	A1
20110149409	Haugholt et al.	Jun 2011	A1
20110150321	Cheong et al.	Jun 2011	A1
20110153248	Gu et al.	Jun 2011	A1
20110157321	Nakajima et al.	Jun 2011	A1
20110157451	Chang	Jun 2011	A1
20110169994	DiFrancesco et al.	Jul 2011	A1
20110176020	Chang	Jul 2011	A1
20110181797	Galstian et al.	Jul 2011	A1
20110193944	Lian et al.	Aug 2011	A1
20110199458	Hayasaka et al.	Aug 2011	A1
20110200319	Kravitz et al.	Aug 2011	A1
20110206291	Kashani et al.	Aug 2011	A1
20110207074	Hall-Holt et al.	Aug 2011	A1
20110211068	Yokota	Sep 2011	A1
20110211077	Nayar et al.	Sep 2011	A1
20110211824	Georgiev et al.	Sep 2011	A1
20110221599	Högasten	Sep 2011	A1
20110221658	Haddick et al.	Sep 2011	A1
20110221939	Jerdev	Sep 2011	A1
20110221950	Oostra et al.	Sep 2011	A1
20110222757	Yeatman, Jr. et al.	Sep 2011	A1
20110228142	Brueckner et al.	Sep 2011	A1
20110228144	Tian et al.	Sep 2011	A1
20110234825	Liu et al.	Sep 2011	A1
20110234841	Akeley et al.	Sep 2011	A1
20110241234	Duparre	Oct 2011	A1
20110242342	Goma et al.	Oct 2011	A1
20110242355	Goma et al.	Oct 2011	A1
20110242356	Aleksic et al.	Oct 2011	A1
20110243428	Das Gupta et al.	Oct 2011	A1
20110255592	Sung et al.	Oct 2011	A1
20110255745	Hodder et al.	Oct 2011	A1
20110255786	Hunter et al.	Oct 2011	A1
20110261993	Weiming et al.	Oct 2011	A1
20110267264	Mccarthy et al.	Nov 2011	A1
20110267348	Lin et al.	Nov 2011	A1
20110273531	Ito et al.	Nov 2011	A1
20110274175	Sumitomo	Nov 2011	A1
20110274366	Tardif	Nov 2011	A1
20110279705	Kuang et al.	Nov 2011	A1
20110279721	McMahon	Nov 2011	A1
20110285701	Chen et al.	Nov 2011	A1
20110285866	Bhrugumalla et al.	Nov 2011	A1
20110285910	Bamji et al.	Nov 2011	A1
20110292216	Fergus et al.	Dec 2011	A1
20110298898	Jung et al.	Dec 2011	A1
20110298917	Yanagita	Dec 2011	A1
20110300929	Tardif et al.	Dec 2011	A1
20110310980	Mathew	Dec 2011	A1
20110316968	Taguchi et al.	Dec 2011	A1
20110317766	Lim et al.	Dec 2011	A1
20120012748	Pain	Jan 2012	A1
20120013748	Stanwood et al.	Jan 2012	A1
20120014456	Martinez Bauza et al.	Jan 2012	A1
20120019530	Baker	Jan 2012	A1
20120019700	Gaber	Jan 2012	A1
20120023456	Sun et al.	Jan 2012	A1
20120026297	Sato	Feb 2012	A1
20120026342	Yu et al.	Feb 2012	A1
20120026366	Golan et al.	Feb 2012	A1
20120026451	Nystrom	Feb 2012	A1
20120026478	Chen et al.	Feb 2012	A1
20120038745	Yu et al.	Feb 2012	A1
20120039525	Tian et al.	Feb 2012	A1
20120044249	Mashitani et al.	Feb 2012	A1
20120044372	Côté et al.	Feb 2012	A1
20120051624	Ando	Mar 2012	A1
20120056982	Katz et al.	Mar 2012	A1
20120057040	Park et al.	Mar 2012	A1
20120062697	Treado et al.	Mar 2012	A1
20120062702	Jiang et al.	Mar 2012	A1
20120062756	Tian et al.	Mar 2012	A1
20120069235	Imai	Mar 2012	A1
20120081519	Goma et al.	Apr 2012	A1
20120086803	Malzbender et al.	Apr 2012	A1
20120105590	Fukumoto et al.	May 2012	A1
20120105654	Kwatra et al.	May 2012	A1
20120105691	Waqas et al.	May 2012	A1
20120113232	Joblove	May 2012	A1
20120113318	Galstian et al.	May 2012	A1
20120113413	Miahczylowicz-Wolski et al.	May 2012	A1
20120114224	Xu et al.	May 2012	A1
20120114260	Takahashi et al.	May 2012	A1
20120120264	Lee et al.	May 2012	A1
20120127275	Von Zitzewitz et al.	May 2012	A1
20120127284	Bar-Zeev et al.	May 2012	A1
20120147139	Li et al.	Jun 2012	A1
20120147205	Lelescu et al.	Jun 2012	A1
20120153153	Chang et al.	Jun 2012	A1
20120154551	Inoue	Jun 2012	A1
20120155830	Sasaki et al.	Jun 2012	A1
20120162374	Markas et al.	Jun 2012	A1
20120163672	McKinnon	Jun 2012	A1
20120163725	Fukuhara	Jun 2012	A1
20120169433	Mullins et al.	Jul 2012	A1
20120170134	Bolis et al.	Jul 2012	A1
20120176479	Mayhew et al.	Jul 2012	A1
20120176481	Lukk et al.	Jul 2012	A1
20120188235	Wu et al.	Jul 2012	A1
20120188341	Klein Gunnewiek et al.	Jul 2012	A1
20120188389	Lin et al.	Jul 2012	A1
20120188420	Black et al.	Jul 2012	A1
20120188634	Kubala et al.	Jul 2012	A1
20120198677	Duparre	Aug 2012	A1
20120200669	Lai et al.	Aug 2012	A1
20120200726	Bugnariu	Aug 2012	A1
20120200734	Tang	Aug 2012	A1
20120206582	DiCarlo et al.	Aug 2012	A1
20120218455	Imai et al.	Aug 2012	A1
20120219236	Ali et al.	Aug 2012	A1
20120224083	Jovanovski et al.	Sep 2012	A1
20120229602	Chen et al.	Sep 2012	A1
20120229628	Ishiyama et al.	Sep 2012	A1
20120237114	Park et al.	Sep 2012	A1
20120249550	Akeley et al.	Oct 2012	A1
20120249750	Izzat et al.	Oct 2012	A1
20120249836	Ali et al.	Oct 2012	A1
20120249853	Krolczyk et al.	Oct 2012	A1
20120250990	Bocirnea	Oct 2012	A1
20120262601	Choi et al.	Oct 2012	A1
20120262607	Shimura et al.	Oct 2012	A1
20120268574	Gidon et al.	Oct 2012	A1
20120274626	Hsieh	Nov 2012	A1
20120287291	McMahon	Nov 2012	A1
20120290257	Hodge et al.	Nov 2012	A1
20120293489	Chen et al.	Nov 2012	A1
20120293624	Chen et al.	Nov 2012	A1
20120293695	Tanaka	Nov 2012	A1
20120307084	Mantzel	Dec 2012	A1
20120307093	Miyoshi	Dec 2012	A1
20120307099	Yahata	Dec 2012	A1
20120314033	Lee et al.	Dec 2012	A1
20120314937	Kim et al.	Dec 2012	A1
20120327222	Ng et al.	Dec 2012	A1
20130002828	Ding et al.	Jan 2013	A1
20130002953	Noguchi et al.	Jan 2013	A1
20130003184	Duparre	Jan 2013	A1
20130010073	Do et al.	Jan 2013	A1
20130016245	Yuba	Jan 2013	A1
20130016885	Tsujimoto	Jan 2013	A1
20130022111	Chen et al.	Jan 2013	A1
20130027580	Olsen et al.	Jan 2013	A1
20130033579	Wajs	Feb 2013	A1
20130033585	Li et al.	Feb 2013	A1
20130038696	Ding et al.	Feb 2013	A1
20130047396	Au et al.	Feb 2013	A1
20130050504	Safaee-Rad et al.	Feb 2013	A1
20130050526	Keelan	Feb 2013	A1
20130057710	McMahon	Mar 2013	A1
20130070060	Chatterjee et al.	Mar 2013	A1
20130076967	Brunner et al.	Mar 2013	A1
20130077859	Stauder et al.	Mar 2013	A1
20130077880	Venkataraman et al.	Mar 2013	A1
20130077882	Venkataraman et al.	Mar 2013	A1
20130083172	Baba	Apr 2013	A1
20130088489	Schmeitz et al.	Apr 2013	A1
20130088637	Duparre	Apr 2013	A1
20130093842	Yahata	Apr 2013	A1
20130100254	Morioka et al.	Apr 2013	A1
20130107061	Kumar et al.	May 2013	A1
20130113888	Koguchi	May 2013	A1
20130113899	Morohoshi et al.	May 2013	A1
20130113939	Strandemar	May 2013	A1
20130120536	Song et al.	May 2013	A1
20130120605	Georgiev et al.	May 2013	A1
20130121559	Hu et al.	May 2013	A1
20130123985	Hirai et al.	May 2013	A1
20130127988	Wang et al.	May 2013	A1
20130128049	Schofield et al.	May 2013	A1
20130128068	Georgiev et al.	May 2013	A1
20130128069	Georgiev et al.	May 2013	A1
20130128087	Georgiev et al.	May 2013	A1
20130128121	Agarwala et al.	May 2013	A1
20130135315	Bares et al.	May 2013	A1
20130135448	Nagumo et al.	May 2013	A1
20130147979	McMahon et al.	Jun 2013	A1
20130155050	Rastogi et al.	Jun 2013	A1
20130162641	Zhang et al.	Jun 2013	A1
20130162980	Kim et al.	Jun 2013	A1
20130169754	Aronsson et al.	Jul 2013	A1
20130176394	Tian et al.	Jul 2013	A1
20130208138	Li et al.	Aug 2013	A1
20130215108	McMahon et al.	Aug 2013	A1
20130215231	Hiramoto et al.	Aug 2013	A1
20130216144	Robinson et al.	Aug 2013	A1
20130222556	Shimada	Aug 2013	A1
20130222656	Kaneko	Aug 2013	A1
20130223759	Nishiyama	Aug 2013	A1
20130229540	Farina et al.	Sep 2013	A1
20130230237	Schlosser et al.	Sep 2013	A1
20130250123	Zhang et al.	Sep 2013	A1
20130250150	Malone et al.	Sep 2013	A1
20130258067	Zhang et al.	Oct 2013	A1
20130259317	Gaddy	Oct 2013	A1
20130265459	Duparre et al.	Oct 2013	A1
20130274596	Azizian et al.	Oct 2013	A1
20130274923	By	Oct 2013	A1
20130278631	Border et al.	Oct 2013	A1
20130286236	Mankowski	Oct 2013	A1
20130293760	Nisenzon et al.	Nov 2013	A1
20130308197	Duparre	Nov 2013	A1
20130321581	El-ghoroury et al.	Dec 2013	A1
20130321589	Kirk et al.	Dec 2013	A1
20130335598	Gustavsson et al.	Dec 2013	A1
20130342641	Morioka et al.	Dec 2013	A1
20140002674	Duparre et al.	Jan 2014	A1
20140002675	Duparre et al.	Jan 2014	A1
20140009586	McNamer et al.	Jan 2014	A1
20140013273	Ng	Jan 2014	A1
20140037137	Broaddus et al.	Feb 2014	A1
20140037140	Benhimane et al.	Feb 2014	A1
20140043507	Wang et al.	Feb 2014	A1
20140059462	Wernersson	Feb 2014	A1
20140076336	Clayton et al.	Mar 2014	A1
20140078333	Miao	Mar 2014	A1
20140079336	Venkataraman et al.	Mar 2014	A1
20140081454	Nuyujukian et al.	Mar 2014	A1
20140085502	Lin et al.	Mar 2014	A1
20140092281	Nisenzon et al.	Apr 2014	A1
20140098266	Nayar et al.	Apr 2014	A1
20140098267	Tian et al.	Apr 2014	A1
20140104490	Hsieh et al.	Apr 2014	A1
20140118493	Sali et al.	May 2014	A1
20140118584	Lee et al.	May 2014	A1
20140125760	Au et al.	May 2014	A1
20140125771	Grossmann et al.	May 2014	A1
20140132810	McMahon	May 2014	A1
20140139642	Ni et al.	May 2014	A1
20140139643	Hogasten et al.	May 2014	A1
20140140626	Cho et al.	May 2014	A1
20140146132	Bagnato et al.	May 2014	A1
20140146201	Knight et al.	May 2014	A1
20140176592	Wilburn et al.	Jun 2014	A1
20140183258	DiMuro	Jul 2014	A1
20140183334	Wang et al.	Jul 2014	A1
20140186045	Poddar et al.	Jul 2014	A1
20140192154	Jeong et al.	Jul 2014	A1
20140192253	Laroia	Jul 2014	A1
20140198188	Izawa	Jul 2014	A1
20140204183	Lee et al.	Jul 2014	A1
20140218546	McMahon	Aug 2014	A1
20140232822	Venkataraman et al.	Aug 2014	A1
20140240528	Venkataraman et al.	Aug 2014	A1
20140240529	Venkataraman et al.	Aug 2014	A1
20140253738	Mullis	Sep 2014	A1
20140267243	Venkataraman et al.	Sep 2014	A1
20140267286	Duparre	Sep 2014	A1
20140267633	Venkataraman et al.	Sep 2014	A1
20140267762	Mullis et al.	Sep 2014	A1
20140267829	McMahon et al.	Sep 2014	A1
20140267890	Lelescu et al.	Sep 2014	A1
20140285675	Mullis	Sep 2014	A1
20140300706	Song	Oct 2014	A1
20140307058	Kirk et al.	Oct 2014	A1
20140307063	Lee	Oct 2014	A1
20140313315	Shoham et al.	Oct 2014	A1
20140321712	Ciurea et al.	Oct 2014	A1
20140333731	Venkataraman et al.	Nov 2014	A1
20140333764	Venkataraman et al.	Nov 2014	A1
20140333787	Venkataraman et al.	Nov 2014	A1
20140340539	Venkataraman et al.	Nov 2014	A1
20140347509	Venkataraman et al.	Nov 2014	A1
20140347748	Duparre	Nov 2014	A1
20140354773	Venkataraman et al.	Dec 2014	A1
20140354843	Venkataraman et al.	Dec 2014	A1
20140354844	Venkataraman et al.	Dec 2014	A1
20140354853	Venkataraman et al.	Dec 2014	A1
20140354854	Venkataraman et al.	Dec 2014	A1
20140354855	Venkataraman et al.	Dec 2014	A1
20140355870	Venkataraman et al.	Dec 2014	A1
20140368662	Venkataraman et al.	Dec 2014	A1
20140368683	Venkataraman et al.	Dec 2014	A1
20140368684	Venkataraman et al.	Dec 2014	A1
20140368685	Venkataraman et al.	Dec 2014	A1
20140368686	Duparre	Dec 2014	A1
20140369612	Venkataraman et al.	Dec 2014	A1
20140369615	Venkataraman et al.	Dec 2014	A1
20140376825	Venkataraman et al.	Dec 2014	A1
20140376826	Venkataraman et al.	Dec 2014	A1
20150002734	Lee	Jan 2015	A1
20150003752	Venkataraman et al.	Jan 2015	A1
20150003753	Venkataraman et al.	Jan 2015	A1
20150009353	Venkataraman et al.	Jan 2015	A1
20150009354	Venkataraman et al.	Jan 2015	A1
20150009362	Venkataraman et al.	Jan 2015	A1
20150015669	Venkataraman et al.	Jan 2015	A1
20150035992	Mullis	Feb 2015	A1
20150036014	Lelescu et al.	Feb 2015	A1
20150036015	Lelescu et al.	Feb 2015	A1
20150042766	Ciurea et al.	Feb 2015	A1
20150042767	Ciurea et al.	Feb 2015	A1
20150042814	Vaziri	Feb 2015	A1
20150042833	Lelescu et al.	Feb 2015	A1
20150049915	Ciurea et al.	Feb 2015	A1
20150049916	Ciurea et al.	Feb 2015	A1
20150049917	Ciurea et al.	Feb 2015	A1
20150055884	Venkataraman et al.	Feb 2015	A1
20150085073	Bruls et al.	Mar 2015	A1
20150085174	Shabtay et al.	Mar 2015	A1
20150091900	Yang et al.	Apr 2015	A1
20150093015	Liang	Apr 2015	A1
20150095235	Dua	Apr 2015	A1
20150098079	Montgomery et al.	Apr 2015	A1
20150104076	Hayasaka	Apr 2015	A1
20150104101	Bryant et al.	Apr 2015	A1
20150122411	Rodda et al.	May 2015	A1
20150124059	Georgiev et al.	May 2015	A1
20150124113	Rodda et al.	May 2015	A1
20150124151	Rodda et al.	May 2015	A1
20150138346	Venkataraman et al.	May 2015	A1
20150146029	Venkataraman et al.	May 2015	A1
20150146030	Venkataraman et al.	May 2015	A1
20150161798	Venkataraman et al.	Jun 2015	A1
20150199793	Venkataraman et al.	Jul 2015	A1
20150199841	Venkataraman et al.	Jul 2015	A1
20150206912	Kanamori et al.	Jul 2015	A1
20150207990	Ford et al.	Jul 2015	A1
20150228081	Kim et al.	Aug 2015	A1
20150235476	McMahon et al.	Aug 2015	A1
20150237329	Venkataraman et al.	Aug 2015	A1
20150243480	Yamada	Aug 2015	A1
20150244927	Laroia et al.	Aug 2015	A1
20150245013	Venkataraman et al.	Aug 2015	A1
20150248744	Hayasaka et al.	Sep 2015	A1
20150254868	Srikanth et al.	Sep 2015	A1
20150256733	Kanamori	Sep 2015	A1
20150264337	Venkataraman et al.	Sep 2015	A1
20150288861	Duparre	Oct 2015	A1
20150296137	Duparre et al.	Oct 2015	A1
20150312455	Venkataraman et al.	Oct 2015	A1
20150317638	Donaldson	Nov 2015	A1
20150326852	Duparre et al.	Nov 2015	A1
20150332468	Hayasaka et al.	Nov 2015	A1
20150373261	Rodda et al.	Dec 2015	A1
20160037097	Duparre	Feb 2016	A1
20160042548	Du et al.	Feb 2016	A1
20160044252	Molina	Feb 2016	A1
20160044257	Venkataraman et al.	Feb 2016	A1
20160057332	Ciurea et al.	Feb 2016	A1
20160065934	Kaza et al.	Mar 2016	A1
20160070030	Fujisawa et al.	Mar 2016	A1
20160163051	Mullis	Jun 2016	A1
20160165106	Duparre	Jun 2016	A1
20160165134	Lelescu et al.	Jun 2016	A1
20160165147	Nisenzon et al.	Jun 2016	A1
20160165212	Mullis	Jun 2016	A1
20160182786	Anderson et al.	Jun 2016	A1
20160191768	Shin et al.	Jun 2016	A1
20160195733	Lelescu et al.	Jul 2016	A1
20160198096	McMahon et al.	Jul 2016	A1
20160209654	Riccomini et al.	Jul 2016	A1
20160210785	Balachandreswaran et al.	Jul 2016	A1
20160216198	Sun et al.	Jul 2016	A1
20160227195	Venkataraman et al.	Aug 2016	A1
20160249001	McMahon	Aug 2016	A1
20160255333	Nisenzon et al.	Sep 2016	A1
20160261844	Kadambi et al.	Sep 2016	A1
20160266284	Duparre et al.	Sep 2016	A1
20160267486	Mitra et al.	Sep 2016	A1
20160267665	Venkataraman et al.	Sep 2016	A1
20160267672	Ciurea et al.	Sep 2016	A1
20160269626	McMahon	Sep 2016	A1
20160269627	McMahon	Sep 2016	A1
20160269650	Venkataraman et al.	Sep 2016	A1
20160269651	Venkataraman et al.	Sep 2016	A1
20160269664	Duparre	Sep 2016	A1
20160309084	Venkataraman et al.	Oct 2016	A1
20160309134	Venkataraman et al.	Oct 2016	A1
20160316140	Nayar et al.	Oct 2016	A1
20160323578	Kaneko et al.	Nov 2016	A1
20170004791	Aubineau et al.	Jan 2017	A1
20170006233	Venkataraman et al.	Jan 2017	A1
20170011405	Pandey	Jan 2017	A1
20170048468	Pain et al.	Feb 2017	A1
20170053382	Lelescu et al.	Feb 2017	A1
20170054901	Venkataraman et al.	Feb 2017	A1
20170070672	Rodda et al.	Mar 2017	A1
20170070673	Lelescu et al.	Mar 2017	A1
20170070753	Kaneko	Mar 2017	A1
20170078568	Venkataraman et al.	Mar 2017	A1
20170085845	Venkataraman et al.	Mar 2017	A1
20170094243	Venkataraman et al.	Mar 2017	A1
20170099465	Mullis et al.	Apr 2017	A1
20170109742	Varadarajan	Apr 2017	A1
20170142405	Shors et al.	May 2017	A1
20170163862	Molina	Jun 2017	A1
20170178363	Venkataraman et al.	Jun 2017	A1
20170178399	Fest	Jun 2017	A1
20170187933	Duparre	Jun 2017	A1
20170188011	Panescu et al.	Jun 2017	A1
20170244960	Ciurea et al.	Aug 2017	A1
20170257562	Venkataraman et al.	Sep 2017	A1
20170268990	Martinello et al.	Sep 2017	A1
20170337682	Liao et al.	Nov 2017	A1
20170365104	McMahon et al.	Dec 2017	A1
20180005012	Aycock et al.	Jan 2018	A1
20180005244	Govindarajan et al.	Jan 2018	A1
20180007284	Venkataraman et al.	Jan 2018	A1
20180013945	Ciurea et al.	Jan 2018	A1
20180024330	Laroia	Jan 2018	A1
20180035057	McMahon et al.	Feb 2018	A1
20180040135	Mullis	Feb 2018	A1
20180048830	Venkataraman et al.	Feb 2018	A1
20180048879	Venkataraman et al.	Feb 2018	A1
20180081090	Duparre et al.	Mar 2018	A1
20180097993	Nayar et al.	Apr 2018	A1
20180100731	Pau	Apr 2018	A1
20180109782	Duparre et al.	Apr 2018	A1
20180124311	Lelescu et al.	May 2018	A1
20180131852	McMahon	May 2018	A1
20180139382	Venkataraman et al.	May 2018	A1
20180177461	Bell et al.	Jun 2018	A1
20180189767	Bigioi	Jul 2018	A1
20180197035	Venkataraman et al.	Jul 2018	A1
20180211402	Ciurea et al.	Jul 2018	A1
20180227511	McMahon	Aug 2018	A1
20180240265	Yang et al.	Aug 2018	A1
20180270473	Mullis	Sep 2018	A1
20180286120	Fleishman et al.	Oct 2018	A1
20180302554	Lelescu et al.	Oct 2018	A1
20180330182	Venkataraman et al.	Nov 2018	A1
20180376122	Park et al.	Dec 2018	A1
20190012768	Tafazoli Bilandi et al.	Jan 2019	A1
20190037116	Molina	Jan 2019	A1
20190037150	Srikanth et al.	Jan 2019	A1
20190043253	Lucas et al.	Feb 2019	A1
20190052792	Baba et al.	Feb 2019	A1
20190057513	Jain et al.	Feb 2019	A1
20190063905	Venkataraman et al.	Feb 2019	A1
20190089947	Venkataraman et al.	Mar 2019	A1
20190098209	Venkataraman et al.	Mar 2019	A1
20190109998	Venkataraman et al.	Apr 2019	A1
20190164341	Venkataraman	May 2019	A1
20190174040	Mcmahon	Jun 2019	A1
20190174077	Mitani et al.	Jun 2019	A1
20190197735	Xiong et al.	Jun 2019	A1
20190215496	Mullis et al.	Jul 2019	A1
20190230348	Ciurea et al.	Jul 2019	A1
20190235138	Duparre et al.	Aug 2019	A1
20190243086	Rodda et al.	Aug 2019	A1
20190244379	Venkataraman	Aug 2019	A1
20190268586	Mullis	Aug 2019	A1
20190289176	Duparre	Sep 2019	A1
20190347768	Lelescu et al.	Nov 2019	A1
20190356863	Venkataraman et al.	Nov 2019	A1
20190362515	Ciurea et al.	Nov 2019	A1
20190364263	Jannard et al.	Nov 2019	A1
20200012119	Pezzaniti et al.	Jan 2020	A1
20200026948	Venkataraman et al.	Jan 2020	A1
20200034998	Schlemper et al.	Jan 2020	A1
20200151894	Jain et al.	May 2020	A1
20200162680	Mitani et al.	May 2020	A1
20200195862	Briggs	Jun 2020	A1
20200204729	Kurita et al.	Jun 2020	A1
20200252597	Mullis	Aug 2020	A1
20200311418	Mahadeswaraswamy et al.	Oct 2020	A1
20200334905	Venkataraman	Oct 2020	A1
20200389604	Venkataraman et al.	Dec 2020	A1
20210042952	Jain et al.	Feb 2021	A1
20210044790	Venkataraman et al.	Feb 2021	A1
20210063141	Venkataraman et al.	Mar 2021	A1
20210081698	Lindeman	Mar 2021	A1
20210089807	Liu et al.	Mar 2021	A1
20210133927	Lelescu et al.	May 2021	A1
20210150748	Ciurea et al.	May 2021	A1

Foreign Referenced Citations (282)

Number	Date	Country
2488005	Apr 2002	CN
1619358	May 2005	CN
1669332	Sep 2005	CN
1727991	Feb 2006	CN
1839394	Sep 2006	CN
1985524	Jun 2007	CN
1992499	Jul 2007	CN
101010619	Aug 2007	CN
101046882	Oct 2007	CN
101064780	Oct 2007	CN
101102388	Jan 2008	CN
101147392	Mar 2008	CN
201043890	Apr 2008	CN
101212566	Jul 2008	CN
101312540	Nov 2008	CN
101427372	May 2009	CN
101551586	Oct 2009	CN
101593350	Dec 2009	CN
101606086	Dec 2009	CN
101785025	Jul 2010	CN
101883291	Nov 2010	CN
102037717	Apr 2011	CN
102164298	Aug 2011	CN
102184720	Sep 2011	CN
102375199	Mar 2012	CN
103004180	Mar 2013	CN
103765864	Apr 2014	CN
104081414	Oct 2014	CN
104508681	Apr 2015	CN
104662589	May 2015	CN
104685513	Jun 2015	CN
104685860	Jun 2015	CN
105409212	Mar 2016	CN
103765864	Jul 2017	CN
104081414	Aug 2017	CN
104662589	Aug 2017	CN
107077743	Aug 2017	CN
107230236	Oct 2017	CN
107346061	Nov 2017	CN
107404609	Nov 2017	CN
104685513	Apr 2018	CN
107924572	Apr 2018	CN
108307675	Jul 2018	CN
104335246	Sep 2018	CN
110044931	Jul 2019	CN
107404609	Feb 2020	CN
107346061	Apr 2020	CN
107230236	Dec 2020	CN
108307675	Dec 2020	CN
107077743	Mar 2021	CN
602011041799.1	Sep 2017	DE
0677821	Oct 1995	EP
0840502	May 1998	EP
1201407	May 2002	EP
1355274	Oct 2003	EP
1734766	Dec 2006	EP
1991145	Nov 2008	EP
1243945	Jan 2009	EP
2026563	Feb 2009	EP
2031592	Mar 2009	EP
2041454	Apr 2009	EP
2072785	Jun 2009	EP
2104334	Sep 2009	EP
2136345	Dec 2009	EP
2156244	Feb 2010	EP
2244484	Oct 2010	EP
0957642	Apr 2011	EP
2336816	Jun 2011	EP
2339532	Jun 2011	EP
2381418	Oct 2011	EP
2386554	Nov 2011	EP
2462477	Jun 2012	EP
2502115	Sep 2012	EP
2569935	Mar 2013	EP
2652678	Oct 2013	EP
2677066	Dec 2013	EP
2708019	Mar 2014	EP
2761534	Aug 2014	EP
2777245	Sep 2014	EP
2867718	May 2015	EP
2873028	May 2015	EP
2888698	Jul 2015	EP
2888720	Jul 2015	EP
2901671	Aug 2015	EP
2973476	Jan 2016	EP
3066690	Sep 2016	EP
2569935	Dec 2016	EP
3201877	Aug 2017	EP
2652678	Sep 2017	EP
3284061	Feb 2018	EP
3286914	Feb 2018	EP
3201877	Mar 2018	EP
2817955	Apr 2018	EP
3328048	May 2018	EP
3075140	Jun 2018	EP
3201877	Dec 2018	EP
3467776	Apr 2019	EP
2708019	Oct 2019	EP
3286914	Dec 2019	EP
2761534	Nov 2020	EP
2888720	Mar 2021	EP
3328048	Apr 2021	EP
2482022	Jan 2012	GB
2708CHENP2014	Aug 2015	IN
361194	Mar 2021	IN
59-025483	Feb 1984	JP
64-037177	Feb 1989	JP
02-285772	Nov 1990	JP
06129851	May 1994	JP
07-015457	Jan 1995	JP
H0756112	Mar 1995	JP
09171075	Jun 1997	JP
09181913	Jul 1997	JP
10253351	Sep 1998	JP
11142609	May 1999	JP
11223708	Aug 1999	JP
11325889	Nov 1999	JP
2000209503	Jul 2000	JP
2001008235	Jan 2001	JP
2001194114	Jul 2001	JP
2001264033	Sep 2001	JP
2001277260	Oct 2001	JP
2001337263	Dec 2001	JP
2002195910	Jul 2002	JP
2002205310	Jul 2002	JP
2002209226	Jul 2002	JP
2002250607	Sep 2002	JP
2002252338	Sep 2002	JP
2003094445	Apr 2003	JP
2003139910	May 2003	JP
2003163938	Jun 2003	JP
2003298920	Oct 2003	JP
2004221585	Aug 2004	JP
2005116022	Apr 2005	JP
2005181460	Jul 2005	JP
2005295381	Oct 2005	JP
2005303694	Oct 2005	JP
2005341569	Dec 2005	JP
2005354124	Dec 2005	JP
2006033228	Feb 2006	JP
2006033493	Feb 2006	JP
2006047944	Feb 2006	JP
2006258930	Sep 2006	JP
2007520107	Jul 2007	JP
2007259136	Oct 2007	JP
2008039852	Feb 2008	JP
2008055908	Mar 2008	JP
2008507874	Mar 2008	JP
2008172735	Jul 2008	JP
2008258885	Oct 2008	JP
2009064421	Mar 2009	JP
2009132010	Jun 2009	JP
2009300268	Dec 2009	JP
2010139288	Jun 2010	JP
2011017764	Jan 2011	JP
2011030184	Feb 2011	JP
2011109484	Jun 2011	JP
2011523538	Aug 2011	JP
2011196741	Oct 2011	JP
2011203238	Oct 2011	JP
2012504805	Feb 2012	JP
2011052064	Mar 2013	JP
2013509022	Mar 2013	JP
2013088414	May 2013	JP
2013526801	Jun 2013	JP
2014519741	Aug 2014	JP
2014521117	Aug 2014	JP
2014535191	Dec 2014	JP
2015022510	Feb 2015	JP
2015522178	Aug 2015	JP
2015534734	Dec 2015	JP
5848754	Jan 2016	JP
2016524125	Aug 2016	JP
6140709	May 2017	JP
2017163550	Sep 2017	JP
2017163587	Sep 2017	JP
2017531976	Oct 2017	JP
2019082853	May 2019	JP
6546613	Jul 2019	JP
2019148453	Sep 2019	JP
2019-220957	Dec 2019	JP
6630891	Dec 2019	JP
2020017999	Jan 2020	JP
6767543	Sep 2020	JP
6767558	Sep 2020	JP
1020050004239	Jan 2005	KR
100496875	Jun 2005	KR
1020110097647	Aug 2011	KR
20140045373	Apr 2014	KR
20170063827	Jun 2017	KR
101824672	Feb 2018	KR
101843994	Mar 2018	KR
101973822	Apr 2019	KR
10-2002165	Jul 2019	KR
10-2111181	May 2020	KR
191151	Jul 2013	SG
11201500910	Oct 2015	SG
200828994	Jul 2008	TW
200939739	Sep 2009	TW
201228382	Jul 2012	TW
I535292	May 2016	TW
1994020875	Sep 1994	WO
2005057922	Jun 2005	WO
2006039906	Apr 2006	WO
2006039906	Apr 2006	WO
2007013250	Feb 2007	WO
2007083579	Jul 2007	WO
2007134137	Nov 2007	WO
2008045198	Apr 2008	WO
2008050904	May 2008	WO
2008108271	Sep 2008	WO
2008108926	Sep 2008	WO
2008150817	Dec 2008	WO
2009073950	Jun 2009	WO
2009151903	Dec 2009	WO
2009157273	Dec 2009	WO
2010037512	Apr 2010	WO
2011008443	Jan 2011	WO
2011026527	Mar 2011	WO
2011046607	Apr 2011	WO
2011055655	May 2011	WO
2011063347	May 2011	WO
2011105814	Sep 2011	WO
2011116203	Sep 2011	WO
2011063347	Oct 2011	WO
2011121117	Oct 2011	WO
2011143501	Nov 2011	WO
2012057619	May 2012	WO
2012057620	May 2012	WO
2012057621	May 2012	WO
2012057622	May 2012	WO
2012057623	May 2012	WO
2012057620	Jun 2012	WO
2012074361	Jun 2012	WO
2012078126	Jun 2012	WO
2012082904	Jun 2012	WO
2012155119	Nov 2012	WO
2013003276	Jan 2013	WO
2013043751	Mar 2013	WO
2013043761	Mar 2013	WO
2013049699	Apr 2013	WO
2013055960	Apr 2013	WO
2013119706	Aug 2013	WO
2013126578	Aug 2013	WO
2013166215	Nov 2013	WO
2014004134	Jan 2014	WO
2014005123	Jan 2014	WO
2014031795	Feb 2014	WO
2014052974	Apr 2014	WO
2014032020	May 2014	WO
2014078443	May 2014	WO
2014130849	Aug 2014	WO
2014131038	Aug 2014	WO
2014133974	Sep 2014	WO
2014138695	Sep 2014	WO
2014138697	Sep 2014	WO
2014144157	Sep 2014	WO
2014145856	Sep 2014	WO
2014149403	Sep 2014	WO
2014149902	Sep 2014	WO
2014150856	Sep 2014	WO
2014153098	Sep 2014	WO
2014159721	Oct 2014	WO
2014159779	Oct 2014	WO
2014160142	Oct 2014	WO
2014164550	Oct 2014	WO
2014164909	Oct 2014	WO
2014165244	Oct 2014	WO
2014133974	Apr 2015	WO
2015048694	Apr 2015	WO
2015048906	Apr 2015	WO
2015070105	May 2015	WO
2015074078	May 2015	WO
2015081279	Jun 2015	WO
2015134996	Sep 2015	WO
2015183824	Dec 2015	WO
2016054089	Apr 2016	WO
2016172125	Oct 2016	WO
2016167814	Oct 2016	WO
2016172125	Apr 2017	WO
2018053181	Mar 2018	WO
2019038193	Feb 2019	WO

Non-Patent Literature Citations (286)

Entry
US 8,957,977 B2, 02/2015, Venkataraman et al. (withdrawn)
Ansari et al., “3-D Face Modeling Using Two Views and a Generic Face Model with Application to 3-D Face Recognition”, Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, Jul. 22, 2003, 9 pgs.
Arnab et al. “Pixelwise instance segmentation with a dynamically instantiated network,” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Apr. 7, 2017, Retrieved on Oct. 26, 2020 from https://openaccess.thecvf.com/content_cvpr_2017/papers/Arnab_Pixelwise_Instance_Segmentation_CFPR_2017_paper.pdf, 11 pages.
Atkinson et al., “Hight-sensitivity analysis of polarization by surface Yeflection,” In: Machine Vision and Applications, Aug. 3, 2018, Retrieved on Oct. 26, 2020 from https://link.springer.com/content/pdf/10.1007/s00138-018-0962-7.pdf, 19 pages.
Aufderheide et al., “A MEMS-based Smart Sensor System for Estimation of Camera Pose for Computer Vision Applications”, Research and Innovation Conference 2011, Jul. 29, 2011, pp. 1-10.
Azorin-Lopez, Jorge, et al. “A Novel Active Imaging Model to Design Visual Systems: A Case of Inspection System for Specular Surfaces.” Sensors 17.7 (2017): 1466, 30 pages.
Bai et al. “Deep watershed transform for instance segmentation,” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nov. 24, 2016, Retrieved on Oct. 26, 2020 from https://openaccess.thecvf.com/content_cvpr_2017/papers/Bai_Deep_Watershed_Transform_CVPR_2017_paper.pdf, 10 pages.
Bajard, Alban, et al. “Non conventional Imaging Systems for 3D Digitization of transparent and/or specular manufactured objects.” QCAV2013, 11th Interntional Conference on Quality Control by Artificial Vision. 2013, 9 pages.
Baker et al., “Limits on Super-Resolution and How to Break Them”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 2002, vol. 24, No. 9, pp. 1167-1183.
Banz et al., “Real-Time Semi-Global Matching Disparity Estimation on the GPU”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 2002, vol. 24, No. 9, pp. 1167-1183.
Barnes, Bryan M., et al. “Enhancing 9 nm Node Dense Patterned Defect Optical Inspection using Polarization, Angle, and Focus.” Metrology, Inspection, and Process Control for Microlithography XXVII. vol. 8681. International Society for Optics and Photonics, 2013, 8 pages.
Barron et al., “Intrinsic Scene Properties from a Single RGB-D Image”, 2013 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2013, Portland, OR, USA, pp. 17-24.
Bennett et al., “Multispectral Bilateral Video Fusion”, Computer Graphics (ACM SIGGRAPH Proceedings), Jul. 25, 2006, published Jul. 30, 2006, 1 pg.
Bennett et al., “Multispectral Video Fusion”, Computer Graphics (ACM SIGGRAPH Proceedings), Jul. 25, 2006, published Jul. 30, 2006, 1 pg.
Berretti et al., “Face Recognition by Super-Resolved 3D Models from Consumer Depth Cameras”, IEEE Transactions on Information Forensics and Security, vol. 9, No. 9, Sep. 2014, pp. 1436-1448.
Bertalmio et al., “Image Inpainting”, Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 2000, ACM Pres/Addison-Wesley Publishing Co., pp. 417-424.
Bertero et al., “Super-resolution in computational imaging”, Micron, Jan. 1, 2003, vol. 34, Issues 6-7, 17 pgs.
Bishop et al., “Full-Resolution Depth Map Estimation from an Aliased Plenoptic Light Field”, ACCV Nov. 8, 2010, Part II, LNCS 6493, pp. 186-200.
Bishop et al., “Light Field Superresolution”, Computational Photography (ICCP), 2009 IEEE International Conference, Conference Date Apr. 16-17, published Jan. 26, 2009, 9 pgs.
Bishop et al., “The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution”, IEEE Transactions on Pattern Analysis and Machine Intelligence, May 2012, vol. 34, No. 5, published Aug. 18, 2011, pp. 972-986.
Blanz et al., “A Morphable Model for the Synthesis of 3D Faces”, In Proceedings of ACM SIGGRAPH 1999, Jul. 1, 1999, pp. 187-194.
Borman, “Topics in Multiframe Superresolution Restoration”, Thesis of Sean Borman, Apr. 2004, 282 pgs.
Borman et al., “Image Sequence Processing”, Dekker Encyclopedia of Optical Engineering, Oct. 14, 2002, 81 pgs.
Borman et al., “Linear models for multi-frame super-resolution restoration under non-affine registration and spatially varying PSF”, Proc. SPIE, May 21, 2004, vol. 5299, 12 pgs.
Borman et al., “Simultaneous Multi-Frame MAP Super-Resolution Video Enhancement Using Spatio-Temporal Priors”, Image Processing, 1999, ICIP 99 Proceedings, vol. 3, pp. 469-473.
Borman et al., “Super-Resolution from Image Sequences—A Review”, Circuits & Systems, 1998, pp. 374-378.
Borman et al., “Nonlinear Prediction Methods for Estimation of Clique Weighting Parameters in NonGaussian Image Models”, Proc. SPIE, Sep. 22, 1998, vol. 3459, 9 pgs.
Borman et al., “Block-Matching Sub-Pixel Motion Estimation from Noisy, Under-Sampled Frames—An Empirical Performance Evaluation”, Proc SPIE, Dec. 28, 1998, vol. 3653, 10 pgs.
Borman et al., “Image Resampling and Constraint Formulation for Multi-Frame Super-Resolution Restoration”, Proc SPIE, Dec. 28, 1998, vol. 3653, 10 pgs.
Bose et al., “Superresolution and Noise Filtering Using Moving Least Squares”, IEEE Transactions on Image Processing, Aug. 2006, vol. 15, Issue 8, published Jul. 17, 2006, pp. 2239-2248.
Boye et al., “Comparison of Subpixel Image Registration Algorithms”, Proc. of SPIE—IS&T Electronic Imaging, Feb. 3, 2009, vol. 7246, pp. 72460X-1-72460X-9; doi: 10.1117/12.810369.
Bruckner et al., “Thin wafer-level camera lenses inspired by insect compound eyes”, Optics Express, Nov. 22, 2010, vol. 18, No. 24, pp. 24379-24394.
Bruckner et al., “Artificial compound eye applying hyperacuity”, Optics Express, Dec. 11, 2006, vol. 14, No. 25, pp. 12076-12084.
Bruckner et al., “Driving microoptical imaging systems towards miniature camera applications”, Proc. SPIE, Micro-Optics, May 13, 2010, 11 pgs.
Bruges Martelo, Javier, et al. “Paperboard Coating Detection Based on Full-Stokes Imaging Polarimetry.” Sensors 21.1 (2021): 208, 14 pages.
Bryan et al., “Perspective Distortion from Interpersonal Distance Is an Implicit Visual Cue for Social Judgments of Faces”, PLOS One, vol. 7, Issue 9, Sep. 26, 2012, e45301, doi:10.1371/journal.pone.0045301, 9 pgs.
Bulat et al., “How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Mar. 21, 2017.
Cai et al., “3D Deformable Face Tracking with a Commodity Depth Camera”, Proceedings of the European Conference on Computer Vision: Part III, Sep. 5-11, 2010, 14pgs.
Capel, “Image Mosaicing and Super-resolution”, Retrieved on Nov. 10, 2012, Retrieved from the Internet at URL:<http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.226.2643&rep=rep1 &type=pdf>, 2001, 269 pgs.
Caron et al., “Multiple camera types simultaneous stereo calibration, Robotics and Automation (ICRA)”, 2011 IEEE International Conference on, May 1, 2011 (May 1, 2011), pp. 2933-2938.
Carroll et al., “Image Warps for Artistic Perspective Manipulation”, ACM Transactions on Graphics (TOG), vol. 29, No. 4, Jul. 26, 2010, Article No. 127, 9 pgs.
Chan et al., “Investigation of Computational Compound-Eye Imaging System with Super-Resolution Reconstruction”, IEEE, ISASSP, Jun. 19, 2006, pp. 1177-1180.
Chan et al., “Extending the Depth of Field in a Compound-Eye Imaging System with Super-Resolution Reconstruction”, Proceedings—International Conference on Pattern Recognition, Jan. 1, 2006, vol. 3, pp. 623-626.
Chan et al., “Super-resolution reconstruction in a computational compound-eye imaging system”, Multidim. Syst. Sign. Process, published online Feb. 23, 2007, vol. 18, pp. 83-101.
Chen et al., “Interactive deformation of light fields”, Symposium on Interactive 3D Graphics, 2005, pp. 139-146.
Chen et al., “KNN Matting”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 2013, vol. 35, No. 9, pp. 2175-2188.
Chen et al., “KNN matting”, 2012 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 16-21, 2012, Providence, RI, USA, pp. 869-876.
Chen et al., “Image Matting with Local and Nonlocal Smooth Priors”, CVPR '13 Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23, 2013, pp. 1902-1907.
Chen et al., “Human Face Modeling and Recognition Through Multi-View High Resolution Stereopsis”, IEEE Conference on Computer Vision and Pattern Recognition Workshop, Jun. 17-22, 2006, 6 pgs.
Chen, Hua, et al. “Polarization Phase-Based Method for Material Classification in Computer Vision,” International Journal of Computer Vision 28(1), 1996, pp. 73-83.
Collins et al., “An Active Camera System for Acquiring Multi-View Video”, IEEE 2002 International Conference on Image Processing, Date of Conference: Sep. 22-25, 2002, Rochester, NY, 4 pgs.
Cooper et al., “The perceptual basis of common photographic practice”, Journal of Vision, vol. 12, No. 5, Article 8, May 25, 2012, pp. 1-14.
Crabb et al., “Real-time foreground segmentation via range and color imaging”, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA, Jun. 23-28, 2008, pp. 1-5.
Cui, Zhaopeng, et al. “Polarimetric multi-view stereo,” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 1558-1567.
Dainese et al., “Accurate Depth-Map Estimation for 3D Face Modeling”, IEEE European Signal Processing Conference, Sep. 4-8, 2005, 4 pgs.
Debevec et al., “Recovering High Dynamic Range Radiance Maps from Photographs”, Computer Graphics (ACM SIGGRAPH Proceedings), Aug. 16, 1997, 10 pgs.
Do, Minh N. “Immersive Visual Communication with Depth”, Presented at Microsoft Research, Jun. 15, 2011, Retrieved from http://minhdo.ece.illinois.edu/talks/ImmersiveComm.pdf, 42 pgs.
Do et al., Immersive Visual Communication, IEEE Signal Processing Magazine, vol. 28, Issue 1, Jan. 2011, DOI: 10.1109/MSP.2010.939075, Retrieved from: http://minhdo.ece.illinois.edu/publications/ImmerComm_SPM.pdf, pp. 58-66.
Dou et al., “End-to-end 3D face reconstruction with deep neural networks”, arXiv:1704.05020v1, Apr. 17, 2017, 10 pgs.
Drouin et al., “Improving Border Localization of Multi-Baseline Stereo Using Border-Cut”, International Journal of Computer Vision, Jul. 5, 2006, vol. 83, Issue 3, 8 pgs.
Drouin et al., “Fast Multiple-Baseline Stereo with Occlusion”, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05), Ottawa, Ontario, Canada, Jun. 13-16, 2005, pp. 540-547.
Drouin et al., “Geo-Consistency for Wide Multi-Camera Stereo”, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1, Jun. 20-25, 2005, pp. 351-358.
Drulea et al., “Motion Estimation Using the Correlation Transform”, IEEE Transactions on Image Processing, Aug. 2013, vol. 22, No. 8, pp. 3260-3270, first published May 14, 2013.
Duparre et al., “Microoptical artificial compound eyes—from design to experimental verification of two different concepts”, Proc. of SPIE, Optical Design and Engineering II, vol. 5962, Oct. 17, 2005, pp. 59622A-1-59622A-12.
Duparre et al., Novel Optics/Micro-Optics for Miniature Imaging Systems, Proc. of SPIE, Apr. 21, 2006, vol. 6196, pp. 619607-1-619607-15.
Duparre et al., “Micro-optical artificial compound eyes”, Bioinspiration & Biomimetics, Apr. 6, 2006, vol. 1, pp. R1-R16.
Duparre et al., “Artificial compound eye zoom camera”, Bioinspiration & Biomimetics, Nov. 21, 2008, vol. 3, pp. 1-6.
Duparre et al., “Artificial apposition compound eye fabricated by micro-optics technology”, Applied Optics, Aug. 1, 2004, vol. 43, No. 22, pp. 4303-4310.
Duparre et al., “Micro-optically fabricated artificial apposition compound eye”, Electronic Imaging—Science and Technology, Prod. SPIE 5301, Jan. 2004, pp. 25-33.
Duparre et al., “Chirped arrays of refractive ellipsoidal microlenses for aberration correction under oblique incidence”, Optics Express, Dec. 26, 2005, vol. 13, No. 26, pp. 10539-10551.
Duparre et al., “Artificial compound eyes—different concepts and their application to ultra flat image acquisition sensors”, MOEMS and Miniaturized Systems IV, Proc. SPIE 5346, Jan. 24, 2004, pp. 89-100.
Duparre et al., “Ultra-Thin Camera Based on Artificial Apposition Compound Eyes”, 10th Microoptics Conference, Sep. 1-3, 2004, 2 pgs.
Duparre et al., “Microoptical telescope compound eye”, Optics Express, Feb. 7, 2005, vol. 13, No. 3, pp. 889-903.
Duparre et al., “Theoretical analysis of an artificial superposition compound eye for application in ultra flat digital image acquisition devices”, Optical Systems Design, Proc. SPIE 5249, Sep. 2003, pp. 408-418.
Duparre et al., “Thin compound-eye camera”, Applied Optics, May 20, 2005, vol. 44, No. 15, pp. 2949-2956.
Duparre et al., “Microoptical Artificial Compound Eyes—Two Different Concepts for Compact Imaging Systems”, 11th Microoptics Conference, Oct. 30-Nov. 2, 2005, 2 pgs.
Eng et al., “Gaze correction for 3D tele-immersive communication system”, IVMSP Workshop, 2013 IEEE 11th. IEEE, Jun. 10, 2013.
Fanaswala, “Regularized Super-Resolution of Multi-View Images”, Retrieved on Nov. 10, 2012 (Nov. 10, 2012). Retrieved from the Internet at URL:<http://www.site.uottawa.ca/-edubois/theses/Fanaswala_thesis.pdf>, 2009, 163 pgs.
Fang et al., “Volume Morphing Methods for Landmark Based 3D Image Deformation”, SPIE vol. 2710, Proc. 1996 SPIE Intl Symposium on Medical Imaging, Newport Beach, CA, Feb. 10, 1996, pp. 404-415.
Fangmin et al., “3D Face Reconstruction Based on Convolutional Neural Network”, 2017 10th International Conference on Intelligent Computation Technology and Automation, Oct. 9-10, 2017, Changsha, China.
Farrell et al., “Resolution and Light Sensitivity Tradeoff with Pixel Size”, Proceedings of the SPIE Electronic Imaging 2006 Conference, Feb. 2, 2006, vol. 6069, 8 pgs.
Farsiu et al., “Advances and Challenges in Super-Resolution”, International Journal of Imaging Systems and Technology, Aug. 12, 2004, vol. 14, pp. 47-57.
Farsiu et al., “Fast and Robust Multiframe Super Resolution”, IEEE Transactions on Image Processing, Oct. 2004, published Sep. 3, 2004, vol. 13, No. 10, pp. 1327-1344.
Farsiu et al., “Multiframe Demosaicing and Super-Resolution of Color Images”, IEEE Transactions on Image Processing, Jan. 2006, vol. 15, No. 1, date of publication Dec. 12, 2005, pp. 141-159.
Fechteler et al., Fast and High Resolution 3D Face Scanning, IEEE International Conference on Image Processing, Sep. 16-Oct. 19, 2007, 4 pgs.
Fecker et al., “Depth Map Compression for Unstructured Lumigraph Rendering”, Proc. SPIE 6077, Proceedings Visual Communications and Image Processing 2006, Jan. 18, 2006, pp. 60770B-1-60770B-8.
Feris et al., “Multi-Flash Stereopsis: Depth Edge Preserving Stereo with Small Baseline Illumination”, IEEE Trans on PAMI, 2006, 31 pgs.
Fife et al., “A 3D Multi-Aperture Image Sensor Architecture”, Custom Integrated Circuits Conference, 2006, CICC '06, IEEE, pp. 281-284.
Fife et al., “A 3MPixel Multi-Aperture Image Sensor with 0.7Mu Pixels in 0.11Mu CMOS”, ISSCC 2008, Session 2, Image Sensors & Technology, 2008, pp. 48-50.
Fischer et al., “Optical System Design”, 2nd Edition, SPIE Press, Feb. 14, 2008, pp. 49-58.
Fischer et al., “Optical System Design”, 2nd Edition, SPIE Press, Feb. 14, 2008, pp. 191-198.
Garg et al., “Unsupervised CNN for Single View Depth Estimation Geometry to the Rescue”, In European Conference on Computer Vision, Springer, Cham, Jul. 2016, 16 pgs.
Gastal et al., “Shared Sampling for Real-Time Alpha Matting”, Computer Graphics Forum, EUROGRAPHICS 2010, vol. 29, Issue 2, May 2010, pp. 575-584.
Georgeiv et al., “Light Field Camera Design for Integral View Photography”, Adobe Systems Incorporated, Adobe Technical Report, 2003, 13 pgs.
Georgiev et al., “Light-Field Capture by Multiplexing in the Frequency Domain”, Adobe Systems Incorporated, Adobe Technical Report, 2003, 13 pgs.
Godard et al., “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 14 pgs.
Goldman et al., “Video Object Annotation, Navigation, and Composition”, In Proceedings of UIST 2008, Oct. 19-22, 2008, Monterey CA, USA, pp. 3-12.
Goodfellow et al., “Generative Adversarial Nets, 2014. Generative adversarial nets”, In Advances in Neural Information Processing Systems (pp. 2672-2680).
Gortler et al., “The Lumigraph”, In Proceedings of SIGGRAPH 1996, published Aug. 1, 1996, pp. 43-54.
Gupta et al., “Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images”, 2013 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2013, Portland, OR, USA, pp. 564-571.
Hacohen et al., “Non-Rigid Dense Correspondence with Applications for Image Enhancement”, ACM Transactions on Graphics, vol. 30, No. 4, Aug. 7, 2011, 9 pgs.
Hamilton, “JPEG File Interchange Format, Version 1.02”, Sep. 1, 1992, 9 pgs.
Hardie, “A Fast Image Super-Algorithm Using an Adaptive Wiener Filter”, IEEE Transactions on Image Processing, Dec. 2007, published Nov. 19, 2007, vol. 16, No. 12, pp. 2953-2964.
Hasinoff et al., “Search-and-Replace Editing for Personal Photo Collections”, 2010 International Conference: Computational Photography (ICCP) Mar. 2010, pp. 1-8.
He et al. Mask r-cnn, In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Hernandez et al., “Laser Scan Quality 3-D Face Modeling Using a Low-Cost Depth Camera”, 20th European Signal Processing Conference, Aug. 27-31, 2012, Bucharest, Romania, pp. 1995-1999.
Hernandez-Lopez et al., “Detecting objects using color and depth segmentation with Kinect sensor”, Procedia Technology, vol. 3, Jan. 1, 2012, pp. 196-204, XP055307680, ISSN: 2212-0173, DOI: 10.1016/j.protcy.2012.03.021.
Higo et al., “A Hand-held Photometric Stereo Camera for 3-D Modeling”, IEEE International Conference on Computer Vision, 2009, pp. 1234-1241.
Hirschmuller, “Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, Jun. 20-26, 2005, 8 pgs.
Hirschmuller et al., “Memory Efficient Semi-Global Matching, ISPRS Annals of the Photogrammetry”, Remote Sensing and Spatial Information Sciences, vol. I-3, 2012, XXII ISPRS Congress, Aug. 25-Sep. 1, 2012, Melbourne, Australia, 6 pgs.
Holoeye Photonics AG, “Spatial Light Modulators”, Oct. 2, 2013, Brochure retrieved from https://web.archive.org/web/20131002061028/http://holoeye.com/wp-content/uploads/Spatial_Light_Modulators.pdf on Oct. 13, 2017, 4 pgs.
Holoeye Photonics AG, “Spatial Light Modulators”, Sep. 18, 2013, retrieved from https://web.archive.org/web/20130918113140/http://holoeye.com/spatial-light-modulators/ on Oct. 13, 2017, 4 pgs.
Holoeye Photonics AG, “LC 2012 Spatial Light Modulator (transmissive)”, Sep. 18, 2013, retrieved from https://web.archive.org/web/20130918151716/http://holoeye.com/spatial-light-modulators/lc-2012-spatial-light-modulator/ on Oct. 20, 2017, 3 pgs.
Horisaki et al., “Superposition Imaging for Three-Dimensionally Space-Invariant Point Spread Functions”, Applied Physics Express, Oct. 13, 2011, vol. 4, pp. 112501-1-112501-3.
Horisaki et al., “Irregular Lens Arrangement Design to Improve Imaging Performance of Compound-Eye Imaging Systems”, Applied Physics Express, Jan. 29, 2010, vol. 3, pp. 022501-1-022501-3.
Horn et al., “LightShop: Interactive Light Field Manipulation and Rendering”, In Proceedings of I3D, Jan. 1, 2007, pp. 121-128.
Hossain et al., “Inexpensive Construction of a 3D Face Model from Stereo Images”, IEEE International Conference on Computer and Information Technology, Dec. 27-29, 2007, 6 pgs.
Hu et al., “A Quantitative Evaluation of Confidence Measures for Stereo Vision”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, Issue 11, Nov. 2012, pp. 2121-2133.
Humenberger er al., “A Census-Based Stereo Vision Algorithm Using Modified Semi-Global Matching and Plane Fitting to Improve Matching Quality”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Jun. 13-18, 2010, San Francisco, CA, 8 pgs.
Isaksen et al., “Dynamically Reparameterized Light Fields”, In Proceedings of SIGGRAPH 2000, 2000, pp. 297-306.
Izadi et al., “KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera”, UIST'11, Oct. 16-19, 2011, Santa Barbara, CA, pp. 559-568.
Jackson et al., “Large Post 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression”, arXiv: 1703.07834v2, Sep. 8, 2017, 9 pgs.
Janoch et al., “A category-level 3-D object dataset: Putting the Kinect to work”, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Nov. 6-13, 2011, Barcelona, Spain, pp. 1168-1174.
Jarabo et al., “Efficient Propagation of Light Field Edits”, In Proceedings of SIACG 2011, 2011, pp. 75-80.
Jiang et al., “Panoramic 3D Reconstruction Using Rotational Stereo Camera with Simple Epipolar Constraints”, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), vol. 1, Jun. 17-22, 2006, New York, NY, USA, pp. 371-378.
Joshi, Color Calibration for Arrays of Inexpensive Image Sensors, Mitsubishi Electric Research Laboratories, Inc., TR2004-137, Dec. 2004, 6 pgs.
Joshi et al., “Synthetic Aperture Tracking: Tracking Through Occlusions”, ICCV IEEE 11th International Conference on Computer Vision; Publication [online]. Oct. 2007 [retrieved Jul. 28, 2014]. Retrieved from the Internet: <URL: http:l/ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4409032&isnumber=4408819>, pp. 1-8.
Jourabloo, “Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting”, I CCV IEEE 11th International Conference on Computer Vision; Publication [online]. Oct. 2007 [retrieved Jul. 28, 2014]. Retrieved from the Internet: <URL: http:l/ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4409032&isnumber=4408819>; pp. 1-8.
Kadambi et al. “Polarized 3d: High-quality depth sensing with polarization cues.” Proceedings of the IEEE International Conference on Computer Vision. 2015.
Kang et al., “Handling Occlusions in Dense Multi-view Stereo”, Computer Vision and Pattern Recognition, 2001, vol. 1, pp. I-103-I-110.
Keeton, “Memory-Driven Computing”, Hewlett Packard Enterprise Company, Oct. 20, 2016, 45 pgs.
Kim, “Scene Reconstruction from a Light Field”, Master Thesis, Sep. 1, 2010 (Sep. 1, 2010), pp. 1-72.
Kim et al., “Scene reconstruction from high spatio-angular resolution light fields”, ACM Transactions on Graphics (TOG)—SIGGRAPH 2013 Conference Proceedings, vol. 32 Issue 4, Article 73, Jul. 21, 2013, 11 pages.
Kirillov et al., “Instancecut: from edges to instances with multicut,” In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nov. 24, 2016, Retrieved on Oct. 26, 2020 from https://openaccess.thecvf.com/content_cvpr_2017/papers/Kirillov_InstanceCut_From_Edges_CVPR_2017_paper.pdf, 11 pages.
Kitamura et al., “Reconstruction of a high-resolution image on a compound-eye image-capturing system”, Applied Optics, Mar. 10, 2004, vol. 43, No. 8, pp. 1719-1727.
Kittler et al., “3D Assisted Face Recognition: A Survey of 3D Imaging, Modelling, and Recognition Approaches”, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jul. 2005, 7 pgs.
Konolige, Kurt “Projected Texture Stereo”, 2010 IEEE International Conference on Robotics and Automation, May 3-7, 2010, pp. 148-155.
Kotsia et al., “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines”, IEEE Transactions on Image Processing, Jan. 2007, vol. 16, No. 1, pp. 172-187.
Krishnamurthy et al., “Compression and Transmission of Depth Maps for Image-Based Rendering”, Image Processing, 2001, pp. 828-831.
Kubota et al., “Reconstructing Dense Light Field From Array of Multifocus Images for Novel View Synthesis”, IEEE Transactions on Image Processing, vol. 16, No. 1, Jan. 2007, pp. 269-279.
Kutulakos et al., “Occluding Contour Detection Using Affine Invariants and Purposive Viewpoint Control”, Computer Vision and Pattern Recognition, Proceedings CVPR 94, Seattle, Washington, Jun. 21-23, 1994, 8 pgs.
Lai et al., “A Large-Scale Hierarchical Multi-View RGB-D Object Dataset”, Proceedings—IEEE International Conference on Robotics and Automation, Conference Date May 9-13, 2011, 8 pgs., DOI:10.1109/ICRA.201135980382.
Lane et al., “A Survey of Mobile Phone Sensing”, IEEE Communications Magazine, vol. 48, Issue 9, Sep. 2010, pp. 140-150.
Lao et al., “3D template matching for pose invariant face recognition using 3D facial model built with isoluminance line based stereo vision”, Proceedings 15th International Conference on Pattern Recognition, Sep. 3-7, 2000, Barcelona, Spain, pp. 911-916.
Lee, “NFC Hacking: The Easy Way”, Defcon Hacking Conference, 2012, 24 pgs.
Lee et al., “Electroactive Polymer Actuator for Lens-Drive Unit in Auto-Focus Compact Camera Module”, ETRI Journal, vol. 31, No. 6, Dec. 2009, pp. 695-702.
Lee et al., “Nonlocal matting”, CVPR 2011, Jun. 20-25, 2011, pp. 2193-2200.
Lee et al., “Automatic Upright Adjustment of Photographs”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 877-884.
LensVector, “How LensVector Autofocus Works”, 2010, printed Nov. 2, 2012 from http://www.lensvector.com/overview.html, 1 pg.
Levin et al., “A Closed Form Solution to Natural Image Matting”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006, vol. 1, pp. 61-68.
Levin et al., “Spectral Matting”, 2007 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 17-22, 2007, Minneapolis, MN, USA, pp. 1-8.
Levoy, “Light Fields and Computational Imaging”, IEEE Computer Society, Sep. 1, 2006, vol. 39, Issue No. 8, pp. 46-55.
Levoy et al., “Light Field Rendering”, Proc. ADM SIGGRAPH '96, 1996, pp. 1-12.
Li et al., “A Hybrid Camera for Motion Deblurring and Depth Map Super-Resolution”, Jun. 23-28, 2008, IEEE Conference on Computer Vision and Pattern Recognition, 8 pgs. Retrieved from www.eecis.udel.edu/˜jye/lab_research/08/deblur-feng.pdf on Feb. 5, 2014.
Li et al., “Fusing Images with Different Focuses Using Support Vector Machines”, IEEE Transactions on Neural Networks, vol. 15, No. 6, Nov. 8, 2004, pp. 1555-1561.
Lim, “Optimized Projection Pattern Supplementing Stereo Systems”, 2009 IEEE International Conference on Robotics and Automation, May 12-17, 2009, pp. 2823-2829.
Liu et al., “Virtual View Reconstruction Using Temporal Information”, 2012 IEEE International Conference on Multimedia and Expo, 2012, pp. 115-120.
Lo et al., “Stereoscopic 3D Copy & Paste”, ACM Transactions on Graphics, vol. 29, No. 6, Article 147, Dec. 2010, pp. 147:1-147:10.
Ma et al., “Constant Time Weighted Median Filtering for Stereo Matching and Beyond”, ICCV'13 Proceedings of the 2013 IEEE International Conference on Computer Vision, IEEE Computer Society, Washington DC, USA, Dec. 1-8, 2013, 8 pgs.
Martinez et al., “Simple Telemedicine for Developing Regions: Camera Phones and Paper-Based Microfluidic Devices for Real-Time, Off-Site Diagnosis”, Analytical Chemistry (American Chemical Society), vol. 80, No. 10, May 15, 2008, pp. 3699-3707.
McGuire et al., “Defocus video matting”, ACM Transactions on Graphics (TOG)—Proceedings of ACM SIGGRAPH 2005, vol. 24, Issue 3, Jul. 2005, pp. 567-576.
Medioni et al., “Face Modeling and Recognition in 3-D”, Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, 2013, 2 pgs.
Meriaudeau, Fabrice, et al. “Polarization imaging for industrial inspection.” Image Processing: Machine Vision Applications. vol. 6813. International Society for Optics and Photonics, 2008, 11 pages.
Merkle et al., “Adaptation and optimization of coding algorithms for mobile 3DTV”, MobileSDTV Project No. 216503, Nov. 2008, 55 pgs.
Michael et al., “Real-time Stereo Vision: Optimizing Semi-Global Matching”, 2013 IEEE Intelligent Vehicles Symposium (IV), IEEE, Jun. 23-26, 2013, Australia, 6 pgs.
Milella et al., “3D reconstruction and classification of natural environments by an autonomous vehicle using multi-baseline stereo”, Intelligent Service Robotics, vol. 7, No. 2, Mar. 2, 2014, pp. 79-92.
Min et al., “Real-Time 3D Face Identification from a Depth Camera”, Proceedings of the IEEE International Conference on Pattern Recognition, Nov. 11-15, 2012, 4 pgs.
Mitra et al., “Light Field Denoising, Light Field Superresolution and Stereo Camera Based Refocussing using a GMM Light Field Patch Prior”, Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on Jun. 16-21, 2012, pp. 22-28.
Miyazaki, Daisuke, et al. “Surface normal estimation of black specular objects from multiview polarization images.” Optical Engineering 56.4 (2016): 041303, 18 pages.
Miyazaki, Daisuke, et al. “Polarization-based surface normal estimation of black specular objects from multiple viewpoints.” 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission. IEEE, 2012, 8 pages.
Miyazaki et al. Transparent surface modeling from a pair of polarization images. IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 26, (1):73-82, Jan. 2004.
Morel, Olivier, et al. “Three-Dimensional Inspection of Highly-Reflective Metallic Objects by Polarization Imaging.” Electronic Imaging Newsletter 15.2 (2005): 4.
Morel, Olivier, et al. “Visual Behaviour Based Bio-Inspired Polarization Techniques in Computer Vision and Robotics.” Developing and Applying Biologically-Inspired Vision Systems: Interdisciplinary Concepts. IGI Global, 2013. 243-272.
Morel, O., et al. “Polarization Imaging for 3D Inspection of Highly Reflective Metallic Objects” Optics and Spectroscopy, vol. 101, No. 1, pp. 11-17, (2006).
Moreno-Noguer et al., “Active Refocusing of Images and Videos”, ACM Transactions on Graphics (TOG)—Proceedings of ACM SIGGRAPH 2007, vol. 26, Issue 3, Jul. 2007, 10 pgs.
Muehlebach, “Camera Auto Exposure Control for VSLAM Applications”, Studies on Mechatronics, Swiss Federal Institute of Technology Zurich, Autumn Term 2010 course, 67 pgs.
Nayar, “Computational Cameras: Redefining the Image”, IEEE Computer Society, Aug. 14, 2006, pp. 30-38.
Ng, “Digital Light Field Photography”, Thesis, Jul. 2006, 203 pgs.
Ng et al., “Super-Resolution Image Restoration from Blurred Low-Resolution Images”, Journal of Mathematical Imaging and Vision, 2005, vol. 23, pp. 367-378.
Ng et al., “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR Feb. 2005, Apr. 20, 2005, pp. 1-11.
Nguyen et al., “Image-Based Rendering with Depth Information Using the Propagation Algorithm”, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005, vol. 5, Mar. 23-23, 2005, pp. II-589-II-592.
Nguyen et al., “Error Analysis for Image-Based Rendering with Depth Information”, IEEE Transactions on Image Processing, vol. 18, Issue 4, Apr. 2009, pp. 703-716.
Nishihara, H.K. “PRISM: A Practical Real-Time Imaging Stereo Matcher”, Massachusetts Institute of Technology, A.I. Memo 780, May 1984, 32 pgs.
Nitta et al., “Image reconstruction for thin observation module by bound optics by using the iterative backprojection method”, Applied Optics, May 1, 2006, vol. 45, No. 13, pp. 2893-2900.
Nomura et al., “Scene Collages and Flexible Camera Arrays”, Proceedings of Eurographics Symposium on Rendering, Jun. 2007, 12 pgs.
Park et al., “Super-Resolution Image Reconstruction”, IEEE Signal Processing Magazine, May 2003, pp. 21-36.
Park et al., “Multispectral Imaging Using Multiplexed Illumination”, 2007 IEEE 11th International Conference on Computer Vision, Oct. 14-21, 2007, Rio de Janeiro, Brazil, pp. 1-8.
Park et al., “3D Face Reconstruction from Stereo Video”, First International Workshop on Video Processing for Security, Jun. 7-9, 2006, Quebec City, Canada, 2006, 8 pgs.
Parkkinen et al., “Characteristic Spectra of Munsell Colors”, Journal of the Optical Society of America A, vol. 6, Issue 2, Feb. 1989, pp. 318-322.
Perwass et al., “Single Lens 3D-Camera with Extended Depth-of-Field”, printed from www.raytrix.de, Jan. 22, 2012, 15 pgs.
Pham et al., “Robust Super-Resolution without Regularization”, Journal of Physics: Conference Series 124, Jul. 2008, pp. 1-19.
Philips 3D Solutions, “3D Interface Specifications, White Paper”, Feb. 15, 2008, 2005-2008 Philips Electronics Nederland B.V., Philips 3D Solutions retrieved from www.philips.com/3dsolutions, 29 pgs.
Polight, “Designing Imaging Products Using Reflowable Autofocus Lenses”, printed Nov. 2, 2012 from http://www.polight.no/tunable-polymer-autofocus-lens-html--11.html, 1 pg.
Pouydebasque et al., “Varifocal liquid lenses with integrated actuator, high focusing power and low operating voltage fabricated on 200 mm wafers”, Sensors and Actuators A: Physical, vol. 172, Issue 1, Dec. 2011, pp. 280-286.
Protter et al., “Generalizing the Nonlocal-Means to Super-Resolution Reconstruction”, IEEE Transactions on Image Processing, Dec. 2, 2008, vol. 18, No. 1, pp. 36-51.
Radtke et al., “Laser lithographic fabrication and characterization of a spherical artificial compound eye”, Optics Express, Mar. 19, 2007, vol. 15, No. 6, pp. 3067-3077.
Rahmann, Stefan. “Polarization images: a geometric interpretation for shape analysis.” Proceedings 15th International Conference on Pattern Recognition. ICPR-2000. vol. 3. IEEE, 2000, 5 pages.
Rajan et al., “Simultaneous Estimation of Super Resolved Scene and Depth Map from Low Resolution Defocused Observations”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, No. 9, Sep. 8, 2003, pp. 1-16.
Rander et al., “Virtualized Reality: Constructing Time-Varying Virtual Worlds from Real World Events”, Proc. of IEEE Visualization '97, Phoenix, Arizona, Oct. 19-24, 1997, pp. 277-283, 552.
Ranjan et al., “HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition”, May 11, 2016 (May 11, 2016), pp. 1-16.
Ren et al. End-to-end instance segmentation with recurrent attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6656-6664, 2017.
Rhemann et al., “Fast Cost-Volume Filtering for Visual Correspondence and Beyond”, IEEE Trans. Pattern Anal. Mach. Intell, 2013, vol. 35, No. 2, pp. 504-511.
Rhemann et al., “A perceptually motivated online benchmark for image matting”, 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 20-25, 2009, Miami, FL, USA, pp. 1826-1833.
Robert et al., “Dense Depth Map Reconstruction: A Minimization and Regularization Approach which Preserves Discontinuities”, European Conference on Computer Vision (ECCV), pp. 439-451, (1996).
Robertson et al., “Dynamic Range Improvement Through Multiple Exposures”, In Proc. of the Int. Conf. on Image Processing, 1999, 5 pgs.
Robertson et al., “Estimation-theoretic approach to dynamic range enhancement using multiple exposures”, Journal of Electronic Imaging, Apr. 2003, vol. 12, No. 2, pp. 219-228.
Romera-Paredes et al. Recurrent instance segmentation. In European Conference on Computer Vision, pp. 312-329. Springer, 2016.
Roy et al., “Non-Uniform Hierarchical Pyramid Stereo for Large Images”, Computer and Robot Vision, 2002, pp. 208-215.
Rusinkiewicz et al., “Real-Time 3D Model Acquisition”, ACM Transactions on Graphics (TOG), vol. 21, No. 3, Jul. 2002, pp. 438-446.
Saatci et al., “Cascaded Classification of Gender and Facial Expression using Active Appearance Models”, IEEE, FGR'06, 2006, 6 pgs.
Sajjan, “Learning to See Transparent Objects,” In: Google, Feb. 12, 2020, Retrieved on Oct. 26, 2020 from https://ai.ggogleblog.com/2020/02/learning-to-see-transparent-objects.html, 6 pages.
Sauer et al., “Parallel Computation of Sequential Pixel Updates in Statistical Tomographic Reconstruction”, ICIP 1995 Proceedings of the 1995 International Conference on Image Processing, Date of Conference: Oct. 23-26, 1995, pp. 93-96.
Scharstein et al., “High-Accuracy Stereo Depth Maps Using Structured Light”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), Jun. 2003, vol. 1, pp. 195-202.
Seitz et al., “Plenoptic Image Editing”, International Journal of Computer Vision 48, Conference Date Jan. 7, 1998, 29 pgs., DOI: 10.1109/ICCV.1998.710696 Source: DBLP Conference: Computer Vision, Sixth International Conference.
Shechtman et al., “Increasing Space-Time Resolution in Video”, European Conference on Computer Vision, LNCS 2350, May 28-31, 2002, pp. 753-768.
Shotton et al., “Real-time human pose recognition in parts from single depth images”, CVPR 2011, Jun. 20-25, 2011, Colorado Springs, CO, USA, pp. 1297-1304.
Shum et al., “Pop-Up Light Field: An Interactive Image-Based Modeling and Rendering System”, Apr. 2004, ACM Transactions on Graphics, vol. 23, No. 2, pp. 143-162, Retrieved from http://131.107.65.14/en-us/um/people/jiansun/papers/PopupLightField_TOG.pdf on Feb. 5, 2014.
Shum et al., “A Review of Image-based Rendering Techniques”, Visual Communications and Image Processing 2000, May 2000, 12 pgs.
Sibbing et al., “Markerless reconstruction of dynamic facial expressions”, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshop: Kyoto, Japan, Sep. 27-Oct. 4, 2009, Institute of Electrical and Electronics Engineers, Piscataway, NJ, Sep. 27, 2009 (Sep. 27, 2009), pp. 1778-1785.
Silberman et al., “Indoor segmentation and support inference from RGBD images”, ECCV'12 Proceedings of the 12th European conference on Computer Vision, vol. Part V, Oct. 7 -13, 2012, Florence, Italy, pp. 746-760.
Stober, “Stanford researchers developing 3-D camera with 12,616 lenses”, Stanford Report, Mar. 19, 2008, Retrieved from http://news.stanford.edu/news/2008/march19/camera-031908.html, 5 pgs.
Stollberg et al., “The Gabor superlens as an alternative wafer-level camera approach inspired by superposition compound eyes of nocturnal insects”, Optics Express, Aug. 31, 2009, vol. 17, No. 18, pp. 15747-15759.
Stolz, Christophe, et al. “Short review of polarimetric imaging based method for 3D measurements” Optics, Photonics and Digital Technologies for Imaging Applications IV. vol. 9896. International Society for Optics and Photonics, 2016, 9 pages.
Stolz, Christophe, et al. “Real time polarization imaging of weld pool surface.” Twelfth International Conference on Quality Control by Artificial Vision 2015. vol. 9534. International Society for Optics and Photonics, 2015, 7 pages.
Sun et al., “Image Super-Resolution Using Gradient Profile Prior”, 2008 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2008, 8 pgs.; DOI: 10.1109/CVPR.2008.4587659.
Taguchi et al., “Rendering-Oriented Decoding for a Distributed Multiview Coding System Using a Coset Code”, Hindawi Publishing Corporation, EURASIP Journal on Image and Video Processing, vol. 2009, Article ID 251081, Online: Apr. 22, 2009, 12 pgs.
Takeda et al., “Super-resolution Without Explicit Subpixel Motion Estimation”, IEEE Transaction on Image Processing, Sep. 2009, vol. 18, No. 9, pp. 1958-1975.
Tallon et al., “Upsampling and Denoising of Depth Maps via Joint-Segmentation”, 20th European Signal Processing Conference, Aug. 27-31, 2012, 5 pgs.
Tanida et al., “Thin observation module by bound optics (TOMBO) concept and experimental verification”, Applied Optics, Apr. 10, 2001, vol. 40, No. 11, pp. 1806-1813.
Tanida et al., “Color imaging with an integrated compound imaging system”, Optics Express, Sep. 8, 2003, vol. 11, No. 18, pp. 2109-2117.
Tao et al., “Depth from Combining Defocus and Correspondence Using Light-Field Cameras”, ICCV '13 Proceedings of the 2013 IEEE International Conference on Computer Vision, Dec. 1, 2013, pp. 673-680.
Taylor, “Virtual camera movement: The way of the future?”, American Cinematographer, vol. 77, No. 9, Sep. 1996, pp. 93-100.
Tseng et al., “Automatic 3-D depth recovery from a single urban-scene image”, 2012 Visual Communications and Image Processing, Nov. 27-30, 2012, San Diego, CA, USA, pp. 1-6.
Uchida et al., 3D Face Recognition Using Passive Stereo Vision, IEEE International Conference on Image Processing 2005, Sep. 14, 2005, 4 pgs.
Vaish et al., “Reconstructing Occluded Surfaces Using Synthetic Apertures: Stereo, Focus and Robust Measures”, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), vol. 2, Jun. 17-22, 2006, pp. 2331-2338.
Vaish et al., “Using Plane + Parallax for Calibrating Dense Camera Arrays”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004, 8 pgs.
Vaish et al., “Synthetic Aperture Focusing Using a Shear-Warp Factorization of the Viewing Transform”, IEEE Workshop on A3DISS, CVPR, 2005, 8 pgs.
Van Der Wal et al., “The Acadia Vision Processor”, Proceedings Fifth IEEE International Workshop on Computer Architectures for Machine Perception, Sep. 13, 2000, Padova, Italy, pp. 31-40.
Veilleux, “CCD Gain Lab: The Theory”, University of Maryland, College Park-Observational Astronomy (ASTR 310), Oct. 19, 2006, pp. 1-5 (online], [retrieved on May 13, 2014]. Retrieved from the Internet <URL: http://www.astro.umd.edu/˜veilleux/ASTR310/fall06/ccd_theory.pdf, 5 pgs.
Venkataraman et al., “PiCam: An Ultra-Thin High Performance Monolithic Camera Array”, ACM Transactions on Graphics (TOG), ACM, US, vol. 32, No. 6, 1 Nov. 1, 2013, pp. 1-13.
Vetro et al., “Coding Approaches for End-To-End 3D TV Systems”, Mitsubishi Electric Research Laboratories, Inc., TR2004-137, Dec. 2004, 6 pgs.
Viola et al., “Robust Real-time Object Detection”, Cambridge Research Laboratory, Technical Report Series, Compaq, CRL Jan. 2001, Feb. 2001, Printed from: http://www.hpl.hp.com/techreports/Compaq-DEC/CRL-2001-1.pdf, 30 pgs.
Vuong et al., “A New Auto Exposure and Auto White-Balance Algorithm to Detect High Dynamic Range Conditions Using CMOS Technology”, Proceedings of the World Congress on Engineering and Computer Science 2008, WCECS 2008, Oct. 22-24, 2008, 5 pgs.
Yvain Quéau, Jean-Denis Durou, Jean-François Aujol. Normal Integration: A Survey. 2016. Hal-01334349v4, 19 pages.
Wang, “Calculation of Image Position, Size and Orientation Using First Order Properties”, Dec. 29, 2010, OPTI521 Tutorial, 10 pgs.
Wang et al., “Soft scissors: an interactive tool for realtime high quality matting”, ACM Transactions on Graphics (TOG)—Proceedings of ACM SIGGRAPH 2007, vol. 26, Issue 3, Article 9, Jul. 2007, 6 pg., published Aug. 5, 2007.
Wang et al., “Automatic Natural Video Matting with Depth”, 15th Pacific Conference on Computer Graphics and Applications, PG '07, Oct. 29-Nov. 2, 2007, Maui, HI, USA, pp. 469-472.
Wang et al., “Image and Video Matting: A Survey”, Foundations and Trends, Computer Graphics and Vision, vol. 3, No. 2, 2007, pp. 91-175.
Wang et al., “Facial Feature Point Detection: A Comprehensive Survey”, arXiv: 1410.1037v1, Oct. 4, 2014, 32 pgs.
Wetzstein et al., “Computational Plenoptic Imaging”, Computer Graphics Forum, 2011, vol. 30, No. 8, pp. 2397-2426.
Wheeler et al., “Super-Resolution Image Synthesis Using Projections Onto Convex Sets in the Frequency Domain”, Proc. SPIE, Mar. 11, 2005, vol. 5674, 12 pgs.
Widanagamaachchi et al., “3D Face Recognition from 2D Images: A Survey”, Proceedings of the International Conference on Digital Image Computing: Techniques and Applications, Dec. 1-3, 2008, 7 pgs.
Wieringa et al., “Remote Non-invasive Stereoscopic Imaging of Blood Vessels: First In-vivo Results of a New Multispectral Contrast Enhancement Technology”, Annals of Biomedical Engineering, vol. 34, No. 12, Dec. 2006, pp. 1870-1878, Published online Oct. 12, 2006.
Wikipedia, “Polarizing Filter (Photography)”, retrieved from http://en.wikipedia.org/wiki/Polarizing_filter_(photography) on Dec. 12, 2012, last modified on Sep. 26, 2012, 5 pgs.
Wilburn, “High Performance Imaging Using Arrays of Inexpensive Cameras”, Thesis of Bennett Wilburn, Dec. 2004, 128 pgs.
Wilburn et al., “High Performance Imaging Using Large Camera Arrays”, ACM Transactions on Graphics, Jul. 2005, vol. 24, No. 3, pp. 1-12.
Wilburn et al., “High-Speed Videography Using a Dense Camera Array”, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., vol. 2, Jun. 27-Jul. 2, 2004, pp. 294-301.
Wilburn et al., “The Light Field Video Camera”, Proceedings of Media Processors 2002, SPIE Electronic Imaging, 2002, 8 pgs.
Wippermann et al., “Design and fabrication of a chirped array of refractive ellipsoidal micro-lenses for an apposition eye camera objective”, Proceedings of SPIE, Optical Design and Engineering II, Oct. 15, 2005, pp. 59622C-1-59622C-11.
Wu et al., “A virtual view synthesis algorithm based on image inpainting”, 2012 Third International Conference on Networking and Distributed Computing, Hangzhou, China, Oct. 21-24, 2012, pp. 153-156.
Xu, L. M., et al. “Light source optimization for automatic visual inspection of piston surface defects” The International Journal of Advanced Manufacturing Technology 91.5 (2017): 2245-2256.
Xu, “Real-Time Realistic Rendering and High Dynamic Range Image Display and Compression”, Dissertation, School of Computer Science in the College of Engineering and Computer Science at the University of Central Florida, Orlando, Florida, Fall Term 2005, 192 pgs.
Yang et al., “Superresolution Using Preconditioned Conjugate Gradient Method”, Proceedings of SPIE—The International Society for Optical Engineering, Jul. 2002, 8 pgs.
Yang et al., “A Real-Time Distributed Light Field Camera”, Eurographics Workshop on Rendering (2002), published Jul. 26, 2002, pp. 1-10.
Yang et al., Model-based Head Pose Tracking with Stereovision, Microsoft Research, Technical Report, MSR-TR-2001-102, Oct. 2001, 12 pgs.
Yokochi et al., “Extrinsic Camera Parameter Estimation Based-on Feature Tracking and GPS Data”, 2006, Nara Institute of Science and Technology, Graduate School of Information Science, LNCS 3851, pp. 369-378.
Zbontar et al., Computing the Stereo Matching Cost with a Convolutional Neural Network, CVPR, 2015, pp. 1592-1599.
Zhang et al., “A Self-Reconfigurable Camera Array”, Eurographics Symposium on Rendering, published Aug. 8, 2004, 12 pgs.
Zhang et al., “Depth estimation, spatially variant image registration, and super-resolution using a multi-lenslet camera”, proceedings of SPIE, vol. 7705, Apr. 23, 2010, pp. 770505-770505-8, XP055113797 ISSN: 0277-786X, DOI: 10.1117/12.852171.
Zhang et al., “Spacetime Faces: High Resolution Capture for Modeling and Animation”, ACM Transactions on Graphics, 2004, 11pgs.
Zheng et al., “Balloon Motion Estimation Using Two Frames”, Proceedings of the Asilomar Conference on Signals, Systems and Computers, IEEE, Comp. Soc. Press, US, vol. 2 of 2, Nov. 4, 1991, pp. 1057-1061.
Zhu et al., “Fusion of Time-of-Flight Depth and Stereo for High Accuracy Depth Maps”, 2008 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2008, Anchorage, AK, USA, pp. 1-8.
Zomet et al., “Robust Super-Resolution”, IEEE, 2001, pp. 1-6.
“File Formats Version 6”, Alias Systems, 2004, 40 pgs.
“Light fields and computational photography”, Stanford Computer Graphics Laboratory, Retrieved from: http://graphics.stanford.edu/projects/lightfield/, Earliest publication online: Feb. 10, 1997, 3 pgs.
“Exchangeable image file format for digital still cameras: Exif Version 2.2”_, Japan Electronics and Information Technology Industries Association, Prepared by Technical Standardization Committee on AV & IT Storage Systems and Equipment, JEITA CP-3451, Apr. 2002, Retrieved from: http://www.exif.org/Exif2-2.PDF, 154 pgs.
Written Opinion for International Application No. PCT/US2020/048604, dated Nov. 13, 2020, 8 pages.
Written Opinion for International Application No. PCT/US2020/051243, dated Dec. 9, 2020, 9 pages.
International Search Report and Written Opinion for International Application No. PCT/US20/54641, dated Feb. 17, 2021, 13 pages.
Gruev et al., ““Material Detection with a CCD Polarization Imager,” 2010 IEEE 39th Applied Imagery Pattern Recognition Workshop (AIPR), Oct. 2010, 7 pages”.
International Preliminary Report on Patentability in International Application No. PCT/US20/51243, dated Mar. 15, 2022, 6 pages.
International Search Report and Written Opinion in International Application No. PCT/US20/51243, dated Dec. 9, 2020, 8 pages.
Office Action in German Appln. No. 112020004391.6, dated Jul. 26, 2022, 8 pages (with English translation).
Office Action in Japanese Appln. 2022-517442, dated Aug. 23, 2022, 11 pages (with English translation).
Office Action in Korean Appln. No. 20227012815, dated Aug. 29, 2022, 10 pages (with English translation).
Decision to Grant a Patent in Japanese Appln. No. 2022-517442, dated Mar. 28, 2023, 4 page (with English translation).

Related Publications (1)

	Number	Date	Country
	20220157070 A1	May 2022	US

Provisional Applications (2)

	Number	Date	Country
	62901731	Sep 2019	US
	63001445	Mar 2020	US

Continuations (1)

	Number	Date	Country
Parent	17266054		US
Child	17586666		US

Systems and methods for surface modeling using polarization cues

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

CPC

International Classifications

Disclaimer

Abstract