Systems and methods for generating and using visual datasets for training computer vision models

Abstract

A system for collecting data for training a computer vision model for shape estimation includes: an imaging system configured to capture one or more images; and a processing system including a processor and memory storing instructions that, when executed by the processor, cause the processor to: receive one or more input images from the imaging system; estimate a pose of an object depicted in the one or more images; render a shape estimate from a 3-D model of the object posed in accordance with the pose of the object; and generate a data point of a training dataset, the data point including one or more images based on the one or more input images and a label corresponding to the one or more images, the label including the shape estimate.

Description

FIELD

Aspects of embodiments of the present disclosure relate to systems and methods for generating and using visual datasets for training computer vision models including object pose detection models.

BACKGROUND

In machine learning or statistical learning, large datasets are commonly used to train models to perform predictions or estimations based on statistical patterns found in the datasets. In the case of supervised training, these datasets generally include data samples or data points represented by example inputs and their corresponding ground truth or “labels” (considering the models to operate as mathematical functions the example inputs correspond to the independent variables and the labels correspond to dependent variables).

For example, when applying machine learning in the particular field of computer vision, these datasets may include input images of a variety of different types of objects and corresponding labels such as textual descriptions of the types of objects depicted in the images and/or the locations of these objects within those images (e.g., as defined by bounding boxes or where each pixel is associated with a class of object depicted by that pixel). One example of such a dataset is ImageNet (see, e.g., J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database. IEEE Computer Vision and Pattern Recognition (CVPR), 2009. and Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (*=equal contribution) ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.), which includes images associated with concepts described by multiple words or phrases (on average, about one thousand images for each of about one hundred thousand different concepts). These visual datasets have been useful in training a wide variety of machine learning models such as deep neural networks (e.g., convolutional neural networks) to perform tasks such as image classification and image segmentation. These trained machine learning models for computer vision have been applied in a variety of areas including autonomous vehicles, robotics for manufacturing and logistics processes, detection of abnormalities in medical imaging, and the like.

SUMMARY

Aspects of embodiments of the present disclosure relate to systems and methods for generating and using visual datasets for training computer vision models including object pose detection models.

According to one embodiment of the present disclosure, a system for collecting data for training a computer vision model for shape estimation includes: an imaging system configured to capture one or more images; and a processing system including a processor and memory storing instructions that, when executed by the processor, cause the processor to: receive one or more input images from the imaging system; estimate a pose of an object depicted in the one or more images; render a shape estimate from a 3-D model of the object posed in accordance with the pose of the object; and generate a data point of a training dataset, the data point including one or more images based on the one or more input images and a label corresponding to the one or more images, the label including the shape estimate.

The imaging system may include a polarization camera system, and the one or more input images may include one or more polarization images.

The one or more polarization images may include a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.

The shape estimate may include a surface normals map rendered from the 3-D model posed in accordance with the pose of the object.

The one or more images of the data point may include the one or more polarization images.

The one or more images of the data point may include one or more polarization signatures computed based on the one or more polarization images.

The one or more images of the data point may include one or more surface normals maps computed from the one or more polarization images.

The shape estimate may include a rendered depth map.

The imaging system may include a depth camera system, and the one or more images may include one or more depth maps.

The pose of the object may be estimated based on aligning a shape of the 3-D model with the one or more depth maps.

The processing system may be further configured to estimate the pose of the object using a computer vision model trained to compute shape estimates based on the one or more input images.

The processing system may be further configured to re-train the computer vision model using the training dataset including the data point.

According to one embodiment of the present disclosure, a method for collecting data for training a computer vision model for shape estimation includes: capturing one or more images of a scene using an imaging system; receiving, by a processing system including a processor and memory, the one or more input images from the imaging system; estimating, by the processing system, a pose of an object depicted in the one or more images; rendering, by the processing system, a shape estimate from a 3-D model of the object posed in accordance with the pose of the object; and generating, by the processing system, a data point of a training dataset, the data point including one or more images based on the one or more input images and a label corresponding to the one or more images, the label including the shape estimate.

The imaging system may include a polarization camera system, and the one or more input images may include one or more polarization images.

The one or more polarization images may include a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.

The shape estimate may include a surface normals map rendered from the 3-D model posed in accordance with the pose of the object.

The one or more images of the data point may include the one or more polarization images.

The one or more images of the data point may include one or more polarization signatures computed based on the one or more polarization images.

The one or more images of the data point may include one or more surface normals maps computed from the one or more polarization images.

The shape estimate may include a rendered depth map.

The imaging system may include a depth camera system, and the one or more images may include one or more depth maps.

The pose of the object may be estimated based on aligning a shape of the 3-D model with the one or more depth maps.

The method may further include estimating the pose of the object using a computer vision model trained to compute shape estimates based on the one or more input images.

The method may further include re-training the computer vision model using the training dataset including the data point.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, together with the specification, illustrate exemplary embodiments of the present invention, and, together with the description, serve to explain the principles of the present invention.

FIG. 1A is a schematic diagram depicting a pose estimation system according to one embodiment of the present disclosure.

FIG. 1B is a high-level depiction of the interaction of light with transparent objects and non-transparent (e.g., diffuse and/or reflective) objects.

FIG. 2A is a perspective view of a camera array according to one embodiment of the present disclosure.

FIG. 2B is a cross sectional view of a portion of a camera array according to one embodiment of the present disclosure.

FIG. 2C is a perspective view of a stereo camera array system according to one embodiment of the present disclosure.

FIG. 3 is a flowchart depicting a method for computing six-degree-of-freedom (6-DoF) poses of objects according to some embodiments of the present disclosure.

FIG. 4A is a flow diagram of a process for object level correspondence according to one embodiment.

FIG. 4B is a block diagram of an architecture for instance segmentation and mask generation according to one embodiment.

FIG. 4C is a more detailed flow diagram of a matching algorithm for identifying object-level correspondence for a particular object instance in a first segmentation mask according to one embodiment.

FIG. 5 is a flowchart depicting a method for computing a pose of an object based on dense correspondences according to some embodiments of the present disclosure.

FIG. 6 is a schematic depiction of a 3-D model, depicted in shaded form, posed in accordance with an initial pose estimate and overlaid onto an observed image of a scene, depicted in line drawing form.

FIG. 7A is a block diagram depicting a pipeline for refining an initial pose estimate using dense correspondences according to one embodiment of the present disclosure.

FIG. 7B is a schematic depiction of mappings between observed images and 3-D mesh models based on image-to-object correspondences computed in accordance with some embodiments of the present disclosure.

FIG. 8 is a flowchart depicting a method for generating datasets including images of known objects and corresponding shape estimates according to one embodiment of the present disclosure.

FIG. 9 is a schematic block diagram depicting training a computer vision model using a dataset according to some embodiments of the present disclosure.

FIG. 10 is a schematic block diagram depicting a computer vision model according to some embodiments of the present disclosure.

FIG. 11 is a block diagram of a shape estimator according to one embodiment of the present disclosure.

FIG. 12 is a flowchart of a method for re-training computer vision model according to one embodiment of the present disclosure.

DETAILED DESCRIPTION

In the following detailed description, only certain exemplary embodiments of the present invention are shown and described, by way of illustration. As those skilled in the art would recognize, the invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.

Aspects of embodiments of the present disclosure relate to systems and methods for generating and using visual datasets for training computer vision models including object pose detection models and surface shape detection models. In some embodiments, these visual datasets include polarization raw frames such as images captured using a polarization camera (a camera that has a polarization filter in its optical path) and/or polarization features (e.g., Stokes vectors, degree of linear polarization (DOLP), and angle of linear polarization (AOLP)), which may be computed from polarization raw frames. These images may be associated with ground truth data relating to the shape of objects, such as clean (e.g., low noise or substantially free of noise) and high resolution surface normals maps (e.g., where each pixel or location in the surface normals map identifies the direction of the surface normals or orientation of the depicted surface as a vector in a particular coordinate system, such as a coordinate system defined with respect to the viewpoint) and depth maps (e.g., where each pixel or location in the depth map identifies the distance from the camera to the surface depicted at that pixel, where the depth map may also be interpreted as a point cloud of 3-D coordinates) and such as poses of 3-D models of objects.

Polarization imaging provides information that would not be available to comparative cameras (e.g., imaging modalities that do not include polarization filters and that therefore do not capture information about the polarization of light). This information includes detecting the shape of reflective and transparent objects, determining the surface normals of objects using Fresnel equations, and robustness to specular reflections (e.g., glare). Accordingly, the use of scene polarization information, in the form of polarization images and/or polarization features (e.g., AOLP/DOLP) provides additional information to that can be used by computer vision models to compute more accurate classifications of objects and detections of their locations, poses, and shapes.

Some embodiments of the present disclosure relate to datasets where each data sample includes images of a scene and corresponding ground truth surface normals maps and/or ground truth depth maps, point clouds, or 3-D models of the surfaces of one or more objects in the scene. The images of a scene may include images captured using one or more imaging modalities, including polarization, polarization features, color, infrared, thermal, depth maps (e.g., captured using passive stereo, active stereo with structured light, time of flight, and the like), and the like.

In some embodiments, these images also include surface normals maps computed from the images. In some cases, the surface normals maps are computed from a depth map captured using a depth camera system (e.g., by computing the slope or gradient between neighboring pixels of the captured depth map). In some cases, the surface normals maps are computed using closed form equations (e.g., the Fresnel equations) in accordance with shape-from-polarization (SfP) techniques.

Techniques for computing the shape of objects from polarization information include Polarized 3D (described in, for example, Kadambi, Achuta, et al. “Polarized 3D: High-quality depth sensing with polarization cues.” Proceedings of the IEEE International Conference on Computer Vision. 2015.) which provides deterministic techniques for computing the surface normal field for an object based on a captured polarization signature of an object (e.g., using a polarization camera system) and a coarse approximation of a depth map (e.g., computed using a depth camera system).

While depth maps and Polarized 3D provide routes to computing surface normals maps directly from captured images (e.g., captured depth maps and/or polarization raw frames), the resulting surface normals maps generally exhibit substantial noise or artifacts in accordance with the characteristics of the underlying sensing technique. For example, depth maps captured through stereo depth camera systems may exhibit errors or noise due to ambiguities due to lack of surface texture or ambiguous surface texture, and/or depth resolution limits due to sensor resolution and feature matching constraints, thereby resulting in errors or noise in the surface normals maps computed therefrom. Surface normals maps computed through the direct application of the Fresnel equations may produce artifacts from: ambiguities that arise when determining the azimuth angle of the surface normal; refractive distortions in estimating the zenith angle; non-uniformly in the polarized lighting from the environment; texture copy artifacts when an object has multiple different unique textures; and fronto-parallel surfaces that produce noise in zenith angle estimations when they are close to zero. Furthermore, as the paper by Kadambi et al. shows, obtaining accurate surface normals through polarization is an involved process that has to address the above ambiguities in the surface normal estimations, along with constraints on depth discontinuities among other aspects. Later work by Ba et al. (Ba, Yunhao, et al. “Deep Shape from Polarization.” ECCV. 2020.) followed up on Kadambi et al. by leveraging the physics of polarization through a deep learning network and training the network to learn the relationships between polarization signatures and the surface normal at the point of reflection while disambiguating the estimated normals in the process. This represents a significant improvement over the prior physics-based approach in that the trained network was able to resolve some of these ambiguities resulting in a reduced mean angular error (MAE) in the estimated surface normals. However, some problems remain when dealing with regions of high frequency, increased specularity, shadows and inter-reflections. In addition, the network of Ba is trained using a dataset of collected images in which the surface normals were computed from 3-D scans captured by a structured light 3-D scanner.

In contrast, the ground truth surface normals maps of datasets in accordance with some embodiments of the present disclosure provide clean (e.g., having low noise or being substantially free of noise) shape information of the objects in a scene (e.g., surface normals maps and/or depth maps) that accurately match the shapes of the objects depicted in the corresponding images. In some embodiments, these clean shape estimates are obtained by detecting the poses of known objects in a scene, aligning accurate 3-D models of those known objects based on the detected poses, and rendering the ground truth shape information based on the posed 3-D models. These approaches generally work due to the existence of accurate 3-D models representing the known objects. This is typically possible in the case where the objects are manufactured objects that are substantially uniform in shape and appearance, and where the 3-D model was created as part of the design process in designing the manufactured object and/or designing the manufacturing process for manufacturing the objects (e.g., when creating molds for injection molding or casting of the parts).

As such, datasets generated in accordance with embodiments of the present disclosure provide training data for training computer vision models to compute estimates or predictions of the shapes of objects, where the surface normals maps of the datasets exhibit lower noise and higher accuracy than comparative datasets based on observed or captured data, as opposed to synthetic datasets generated through computer simulations, such as by rending synthetic images and synthetic surface normals maps of a virtual scene using a 3-D graphics engine. In some embodiments, these datasets are used to train computer vision models (e.g., trained statistical models) to generalize from the clean data ground truth data in the training data set and thereby enable the prediction or estimation or inference of the shapes of unknown objects (e.g., objects for which the computer vision system does not have a 3-D model, such as may be the case when the objects are unique, have high variability in shape and appearance, highly diverse, or where accurate 3-D models are otherwise not available to the computer vision system).

Some aspects of embodiments of the present disclosure relate to an integrated system, including imaging hardware and an integrated physics-based deep learning system, that estimates surface normals of known objects (e.g., in a manufacturing assembly line) with very high accuracy. In some embodiments, the imaging hardware implements a multi-view, multi-spectral, and multi-modal approach to image acquisition, and the physics-based deep learning system leverages this additional information to overcome many of the shortcomings of comparative approaches.

Additional aspects of embodiments relate to systems and methods for generating a corpus of data that provides correlations between the various signatures that are captured by imaging hardware (e.g., multi-view, multi-spectral, and multi-modal images) and the final computed six degree-of-freedom (6-DoF) pose, surface normals, and depth estimate of the object in a manner that can be used to train deep learning networks to correctly detect objects and estimate their poses based on the captured signatures.

To provide some context, FIG. 1A is a schematic diagram depicting a pose estimation system according to one embodiment of the present disclosure. As shown in FIG. 1A, a main camera 10 is arranged such that its field of view 12 captures an arrangement 20 of objects 22 resting on a support platform 2 in a scene 1. In the embodiment shown in FIG. 1A, the main camera 10 is located above the support platform (e.g., spaced apart from the objects 22 along the direction of gravity), but embodiments of the present disclosure are not limited thereto—for example, the main camera 10 can be arranged to have a downward angled view of the objects 22.

In some embodiments, one or more support cameras 30 are arranged at different poses or viewpoints around the scene containing the arrangement 20 of objects 22. Accordingly, each of the support cameras 30, e.g., first support camera 30a, second support camera 30b, and third support camera 30c, captures a different view of the objects 22 from a different viewpoint (e.g., a first viewpoint, a second viewpoint, and a third viewpoint, respectively) from one another and a different viewpoint from the main camera 10. The viewpoints may be distinguished from one another in that they have substantially different optical axes, such as optical axes that are not parallel (non-parallel) to one another or that are spaced apart by a large distance if they are parallel to one another.

While FIG. 1A shows three support cameras 30, embodiments of the present disclosure are not limited thereto and may include, for example, at least one support camera 30 and may include more than three support cameras 30. In some embodiments, no support cameras are used and only a single main camera 10 is used from a single viewpoint.

In addition, while the main camera 10 is depicted in FIG. 1A as a stereo camera, embodiments of the present disclosure are not limited thereto, and may be used with, for example, a monocular main camera. As used herein, a stereo camera will be referred to as capturing images from a single viewpoint, as the camera modules of a stereo camera generally have optical axes that are substantially parallel to one another (and may be rectified to synthetically produce such parallel optical axes) and are generally spaced apart along a relatively short baseline to generate a depth map using stereo from a single viewpoint.

A shape estimator 100 according to various embodiments of the present disclosure is configured to compute or estimate shapes and/or poses of the objects 22 based on information captured by the main camera 10 and the support cameras 30. According to various embodiments of the present disclosure, the shape estimator 100 is implemented using one or more processing circuits or electronic circuits configured to perform various operations as described in more detail below. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (AI) accelerator (e.g., a vector processor, which may include vector arithmetic logic units configured efficiently perform operations common to neural networks, such dot products and softmax), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or the like. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented in program instructions that are stored in a non-volatile computer readable memory where, when executed by the electronic circuit (e.g., a CPU, a GPU, an AI accelerator, or combinations thereof), perform the operations described herein to compute a processing output, such as an instance segmentation map and/or 6-DoF poses, from input images 18 (including, for example, polarization raw frames or the underlying images captured by polarization cameras or cameras with polarization filters in their optical paths). The operations performed by the shape estimator 100 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, or the like) or may be allocated between multiple electronic circuits (e.g., multiple GPUs or a CPU in conjunction with a GPU). The multiple electronic circuits may be local to one another (e.g., located on a same die, located within a same package, or located within a same embedded device or computer system) and/or may be remote from one other (e.g., in communication over a network such as a local personal area network such as Bluetooth®, over a local area network such as a local wired and/or wireless network, and/or over wide area network such as the internet, such a case where some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits operating to implement the shape estimator 100 may be referred to herein as a computer or a computer system, which may include memory storing instructions that, when executed by the one or more electronic circuits, implement the systems and methods described herein.

In more detail, the main camera 10 and the support cameras 30 are configured to estimate the shapes and/or poses of objects 22 detected within their fields of view 12 (while FIG. 1A illustrates a field of view 12 for the main camera 10 using dashed lines, the fields of view of the support cameras 30 are not explicitly shown). In the embodiment shown in FIG. 1A, the objects 22 are depicted abstractly as simple three-dimensional solids such as spheres, rectangular prisms, and cylinders. However, embodiments of the present disclosure are not limited thereto and characterization of shape estimators may be performed using any arbitrary object for which a pose with respect to a camera can be clearly defined, including deformable objects mentioned above, such as flex circuits, bags or other pliable containers containing solids, liquids, and/or fluids, flexible tubing, and the like.

In particular, a “pose” refers to the position and orientation of an object with respect to a reference coordinate system. For example, a reference coordinate system may be defined with the main camera 10 at the origin, where the direction along the optical axis of the main camera 10 (e.g., a direction through the center of its field of view 12) is defined as the z-axis of the coordinate system, and the x and y axes are defined to be perpendicular to one another and perpendicular to the z-axis. (Embodiments of the present disclosure are not limited to this particular coordinate system, and a person having ordinary skill in the art would understand that poses can be mathematically transformed to equivalent representations in different coordinate systems.)

Each object 22 may also be associated with a corresponding coordinate system of its own, which is defined with respect to its particular shape. For example, a rectangular prism with sides of different lengths may have a canonical coordinate system defined where the x-axis is parallel to its shortest direction, z-axis is parallel to its longest direction, the y-axis is orthogonal to the x-axis and z-axis, and the origin is located at the centroid of the object 22.

Generally, in a three-dimensional coordinate system, objects 22 have six degrees of freedom—rotation around three axes (e.g., rotation around x-, y-, and z-axes) and translation along the three axes (e.g., translation along x-, y-, and z-axes). For the sake of clarity, symmetries of the objects 22 will not be discussed in detail herein, but may be addressed, for example, by identifying multiple possible poses with respect to different symmetries (e.g., in the case of selecting the positive versus negative directions of the z-axis of a right rectangular prism), or by ignoring some rotational components of the pose (e.g., a right cylinder is rotationally symmetric around its axis).

In some embodiments, it is assumed that a three-dimensional (3-D) model or computer aided design (CAD) model representing a canonical or ideal version of each type of object 22 in the arrangement of objects 20 is available. For example, in some embodiments of the present disclosure, the objects 22 are individual instances of manufactured components that have a substantially uniform appearance from one component to the next. Examples of such manufactured components include screws, bolts, nuts, connectors, and springs, as well as specialty parts such electronic circuit components (e.g., packaged integrated circuits, light emitting diodes, switches, resistors, and the like), laboratory supplies (e.g. test tubes, PCR tubes, bottles, caps, lids, pipette tips, sample plates, and the like), and manufactured parts (e.g., handles, switch caps, light bulbs, and the like). Accordingly, in these circumstances, a CAD model defining the ideal or canonical shape of any particular object 22 in the arrangement 20 may be used to define a coordinate system for the object (e.g., the coordinate system used in the representation of the CAD model).

Based on a reference coordinate system (or camera space, e.g., defined with respect to the pose estimation system) and an object coordinate system (or object space, e.g., defined with respect to one of the objects), the pose of the object may be considered to be a rigid transform (rotation and translation) from object space to camera space. The pose of object 1 in camera space 1 may be denoted as P_c1¹, and the transform from object 1 space to camera space may be represented by the matrix:

$\left[\begin{array}{cccc}{R}_{11}& R_{12}& R_{13}& T_1\\ R_{21}& R_{22}& R_{23}& T_2\\ R_{31}& R_{32}& R_{33}& T_3\\ 0& 0& 0& 1\end{array}\right]$ where the rotation submatrix R:

$R=\left[\begin{array}{ccc}{R}_{11}& R_{12}& R_{13}\\ R_{21}& R_{22}& R_{23}\\ R_{31}& R_{32}& R_{33}\end{array}\right]$ represents rotations along the three axes from object space to camera space, and the translation submatrix T:

$T=\left[\begin{array}{c}{T}_1\\ T_2\\ T_3\end{array}\right]$ represents translations along the three axes from object space to camera space.

If two objects—Object A and Object B—are in the same camera C coordinate frame, then the notation P_CA is used to indicate the pose of Object A with respect to camera C and P_CB is used to indicate the pose of Object B with respect to camera C. For the sake of convenience, it is assumed herein that the poses of objects are represented based on the reference coordinate system, so the poses of objects A and B with respect to camera space C may be denoted P_A and P_B, respectively.

If Object A and Object B are actually the same object, but performed during different pose estimation measurements, and a residual pose P_err or P_AB (P_AB=P_err) is used to indicate a transform from pose P_A to pose P_B, then the following relationship should hold:P_AP_err=P_B  (1)and thereforeP_err=P_A⁻¹P_B  (2)

Ideally, assuming the object has not moved (e.g., translated or rotated) with respect to the main camera 10 between the measurements of pose estimates P_A and P_B, then P_A and P_B should both be the same, and P_err should be the identity matrix (e.g., indicating no error between the poses):

$\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$

In a similar manner, the pose of a particular object can be computed with respect to views from two different cameras. For example, images of Object A captured by a main camera C can be used to compute the pose P_CA of Object A with respect to main camera C. Likewise, images of Object A captured by a first support camera S₁ can be used to compute the pose P_S1A of object A with respect to the support camera S₁. If the relative poses of main camera C and support camera S₁ are known, then the pose P_S1A can be transformed to the coordinate system of the main camera C.

Ideally, assuming that the known relative poses of main camera C and support camera S₁ are accurate and the poses calculated based on the data captured by the two cameras is accurate, then P_CA and P_S1A should both be the same, and P_err should be the identity matrix (e.g., indicating no error between the poses):

$\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$

Differences P_err between the actual measured value as computed based on the estimates computed by the shape estimator 100 and the identity matrix may be considered to be errors:R_err=∥R(P_err)∥  (3)T_err=∥T(P_err)∥  (4)where R_err is the rotation error and T_err is the translation error. The function R( ) converts P_err into an axis-angle where the magnitude is the rotation difference, and the function T( ) extracts the translation component of the pose matrix.

The axis-angle representation from rotation matrix R is given by:

$\begin{array}{cc}Tr\left(R\right)=1+2\cos \theta & \left(5\right)\end{array}$ $\begin{array}{cc}❘\theta ❘=\mathrm{arc}\cos \left(\frac{Tr\left(R\right)-1}{2}\right)& \left(6\right)\end{array}$ where Tr( ) denotes the matrix trace (the sum of the diagonal elements of the matrix), and θ represents the angle of rotation.

Some aspects of embodiments of the present disclosure relate to computing a high accuracy pose estimate of objects 22 in a scene based on a joint estimate of the poses the objects across the main camera 10 and the support cameras 30, and/or the shapes of objects 22 in the scene 1 (e.g., the surface normals or slopes of the objects 22 in the scene and/or the 3-D coordinates of points on the surfaces of the objects), as described in more detail below.

Some aspects of embodiments of the present disclosure also relate to providing information to assist in the control of a robotic arm 24 having an end effector 26 that may be used to grasp and manipulate objects 22. The robotic arm 24, including its end effector 26, may be controlled by a robotic arm controller 28, which, in some embodiments, receives the six-degree-of-freedom poses and/or shapes of objects computed by the shape estimator 100, which may include 3-D models representing various objects 22 in the scene 1, where the 3-D models have configurations that estimate or approximate the configurations of their corresponding real-world objects, noting, for example, that the configuration of portions of the objects 22 that are occluded or otherwise not visible in the fields of view 12 of the main camera 10 and support cameras 30 may be difficult or impossible to estimate with high accuracy.

While the sensor system is generally referred to herein as including a shape estimator 100, embodiments of the present disclosure are not limited to computing shapes and poses (e.g., 6-DoF poses) of objects in a scene and may, instead of or in addition to computing 6-DoF poses, the sensor system, including one or more cameras (e.g., main camera and/or support cameras) and processing circuits may implement generalized vision systems that provide information to controller systems.

For example, a processing pipeline may include receiving images captured by sensor devices (e.g., main cameras 10 and support cameras 30) and outputting control commands for controlling a robot arm, where the processing pipeline is trained, in an end-to-end manner, based on training data that includes sensor data as input and commands for controlling the robot arm (e.g., a destination pose for the end effector 26 of the robotic arm 24) as the labels for the input training data.

As shown in FIG. 1A, the 6-DoF poses computed by the shape estimator 100 may be supplied to a renderer, which is configured to compute or render images of 3-D models posed in a virtual scene in accordance with the poses computed by the shape estimator 100, where the images are rendered from the viewpoints of virtual cameras that may correspond to the viewpoints of the main camera 10 and/or one or more support cameras 30 on the actual scene 1, such that the rendered images correspond to the estimated views of the objects 22 detected in the scene 1 and thereby provide estimates of the shape of the object, including in the case where the rendered images include rendered surface normals maps, as described in more detail below.

Sensing Hardware

In the embodiment shown in FIG. 1A, the pose estimation system includes a main camera 10 and one or more support cameras 30. In some embodiments of the present disclosure, the main camera 10 includes a stereo camera. Examples of stereo cameras include camera systems that have at least two monocular cameras spaced apart from each other along a baseline, where the monocular cameras have overlapping fields of view and optical axes that are substantially parallel to one another. While embodiments of the present disclosure will be presented herein in embodiments where the main camera 10 and the support cameras 30 are passive cameras (e.g., that are not connected to a dedicated light projector and that instead use ambient lighting or other light sources), embodiments of the present disclosure are not limited thereto and may also include circumstances where one or more active light projector are included in the camera system, thereby forming an active camera system, where the active light projector may be configured to project structured light or a pattern onto the scene. The support cameras 30 may be stereo cameras, monocular cameras, or combinations thereof (e.g., some stereo support cameras and some monocular support cameras). In some embodiments, the main camera 10 and/or one or more support cameras 30 may include one or more time-of-flight depth camera systems.

The main camera 10 and the support cameras 30 may use the same imaging modalities or different imaging modalities, and each of the main camera 10 and support cameras 30 may capture images using one or more different imaging modalities. Examples of imaging modalities include monochrome, color, infrared, ultraviolet, thermal, polarization, and combinations thereof (e.g., polarized color, polarized infrared, unpolarized ultraviolet, etc.).

The interaction between light and transparent objects is rich and complex, but the material of an object determines its transparency under visible light. For many transparent household objects, the majority of visible light passes straight through and a small portion (˜4% to ˜8%, depending on the refractive index) is reflected. This is because light in the visible portion of the spectrum has insufficient energy to excite atoms in the transparent object. As a result, the texture (e.g., appearance) of objects behind the transparent object (or visible through the transparent object) dominate the appearance of the transparent object. For example, when looking at a transparent glass cup or tumbler on a table, the appearance of the objects on the other side of the tumbler (e.g., the surface of the table) generally dominate what is seen through the cup. This property leads to some difficulties when attempting to detect surface characteristics of transparent objects such as glass windows and glossy, transparent layers of paint, based on intensity images alone:

FIG. 1B is a high-level depiction of the interaction of light with transparent objects and non-transparent (e.g., diffuse and/or reflective) objects. As shown in FIG. 1B, in some embodiments the main camera 10 includes a polarization camera 11 that captures polarization raw frames of a scene that includes a transparent object 41 in front of an opaque background object 42. A light ray 43 hitting the image sensor 14 of the polarization camera contains polarization information from both the transparent object 41 and the background object 42. The small fraction of reflected light 44 from the transparent object 41 is heavily polarized, and thus has a large impact on the polarization measurement, in contrast to the light 45 reflected off the background object 42 and passing through the transparent object 41.

Similarly, a light ray hitting the surface of an object may interact with the shape of the surface in various ways. For example, a surface with a glossy paint may behave substantially similarly to a transparent object in front of an opaque object as shown in FIG. 1B, where interactions between the light ray and a transparent or translucent layer (or clear coat layer) of the glossy paint causes the light reflecting off of the surface to be polarized based on the characteristics of the transparent or translucent layer (e.g., based on the thickness and surface normals of the layer), which are encoded in the light ray hitting the image sensor. Similarly, as discussed in more detail below with respect to shape from polarization (SfP) theory, variations in the shape of the surface (e.g., direction of the surface normals) may cause significant changes in the polarization of light reflected by the surface of the object. For example, smooth surfaces may generally exhibit the same polarization characteristics throughout, but a scratch or a dent in the surface changes the direction of the surface normals in those areas, and light hitting scratches or dents may be polarized, attenuated, or reflected in ways different than in other portions of the surface of the object. Models of the interactions between light and matter generally consider three fundamentals: geometry, lighting, and material. Geometry is based on the shape of the material. Lighting includes the direction and color of the lighting. Material can be parameterized by the refractive index or angular reflection/transmission of light. This angular reflection is known as a bi-directional reflectance distribution function (BRDF), although other functional forms may more accurately represent certain scenarios. For example, the bidirectional subsurface scattering distribution function (BSSRDF) would be more accurate in the context of materials that exhibit subsurface scattering (e.g. marble or wax).

A light ray 43 hitting the image sensor 14 of a polarization camera has three measurable components: the intensity of light (intensity image/I), the percentage or proportion of light that is linearly polarized (degree of linear polarization/DOLP/p), and the direction of that linear polarization (angle of linear polarization/AOLP/p). These properties encode information about the surface curvature and material of the object being imaged, which can be used by the shape estimator 100 to detect transparent objects, as described in more detail below. In some embodiments, by using one or more polarization cameras, the shape estimator 100 can detect the shapes of optically challenging objects (e.g., that include surfaces made of materials having optically challenging properties such as transparency, reflectivity, or dark matte surfaces) based on similar polarization properties of light passing through translucent objects and/or light interacting with multipath inducing objects or by non-reflective objects (e.g., matte black objects).

In more detail, the polarization camera 11 may further includes a polarizer or polarizing filter or polarization mask 16 placed in the optical path between the scene 1 and the image sensor 14. According to various embodiments of the present disclosure, the polarizer or polarization mask 16 is configured to enable the polarization camera 11 to capture images of the scene 1 with the polarizer set at various specified angles (e.g., at 45° rotations or at 60° rotations or at non-uniformly spaced rotations).

As one example, FIG. 1B depicts an embodiment where the polarization mask 16 is a polarization mosaic aligned with the pixel grid of the image sensor 14 in a manner similar to a red-green-blue (RGB) color filter (e.g., a Bayer filter) of a color camera. In a manner similar to how a color filter mosaic filters incoming light based on wavelength such that each pixel in the image sensor 14 receives light in a particular portion of the spectrum (e.g., red, green, or blue) in accordance with the pattern of color filters of the mosaic, a polarization mask 16 using a polarization mosaic filters light based on linear polarization such that different pixels receive light at different angles of linear polarization (e.g., at 0°, 45°, 90°, and 135°, or at 0°, 60° degrees, and 120°). Accordingly, the polarization camera 11 using a polarization mask 16 such as that shown in FIG. 1B is capable of concurrently or simultaneously capturing light at four different linear polarizations. One example of a polarization camera is the Blackfly® S Polarization Camera produced by FLIR® Systems, Inc. of Wilsonville, Oregon.

While the above description relates to some possible implementations of a polarization camera using a polarization mosaic, embodiments of the present disclosure are not limited thereto and encompass other types of polarization cameras that are capable of capturing images at multiple different polarizations. For example, the polarization mask 16 may have fewer than four polarizations or more than four different polarizations, or may have polarizations at different angles than those stated above (e.g., at angles of polarization of: 0°, 60°, and 120° or at angles of polarization of 0°, 30°, 60°, 90°, 120°, and 150°). As another example, the polarization mask 16 may be implemented using an electronically controlled polarization mask, such as an electro-optic modulator (e.g., may include a liquid crystal layer), where the polarization angles of the individual pixels of the mask may be independently controlled, such that different portions of the image sensor 14 receive light having different polarizations. As another example, the electro-optic modulator may be configured to transmit light of different linear polarizations when capturing different frames, e.g., so that the camera captures images with the entirety of the polarization mask set to, sequentially, to different linear polarizer angles (e.g., sequentially set to: 0 degrees; 45 degrees; 90 degrees; or 135 degrees). As another example, the polarization mask 16 may include a polarizing filter that rotates mechanically, such that different polarization raw frames are captured by the polarization camera 11 with the polarizing filter mechanically rotated with respect to the lens 18 to transmit light at different angles of polarization to image sensor 14. Furthermore, while the above examples relate to the use of a linear polarizing filter, embodiments of the present disclosure are not limited thereto and also include the use of polarization cameras that include circular polarizing filters (e.g., linear polarizing filters with a quarter wave plate). Accordingly, in various embodiments of the present disclosure, a polarization camera uses a polarizing filter to capture multiple polarization raw frames at different polarizations of light, such as different linear polarization angles and different circular polarizations (e.g., handedness).

As a result, the polarization camera 11 captures multiple input images (or polarization raw frames) of the scene including the surfaces of the objects 22. In some embodiments, each of the polarization raw frames corresponds to an image taken behind a polarization filter or polarizer at a different angle of polarization ϕ_pol (e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). Each of the polarization raw frames is captured from substantially the same pose with respect to the scene 1 (e.g., the images captured with the polarization filter at 0 degrees, 45 degrees, 90 degrees, or 135 degrees are all captured by a same polarization camera 11 located at a same location and orientation), as opposed to capturing the polarization raw frames from disparate locations and orientations with respect to the scene. The polarization camera 11 may be configured to detect light in a variety of different portions of the electromagnetic spectrum, such as the human-visible portion of the electromagnetic spectrum, red, green, and blue portions of the human-visible spectrum, as well as invisible portions of the electromagnetic spectrum such as infrared and ultraviolet.

FIG. 2A is a perspective view of a camera array 10′ according to one embodiment of the present disclosure. FIG. 2B is a cross sectional view of a portion of a camera array 10′ according to one embodiment of the present disclosure. Some aspects of embodiments of the present disclosure relate to a camera array in which multiple cameras (e.g., cameras having different imaging modalities and/or sensitivity to different spectra) are arranged adjacent to one another and in an array and may be controlled to capture images in a group (e.g., a single trigger may be used to control all of the cameras in the system to capture images concurrently or substantially simultaneously). In some embodiments, the individual cameras are arranged such that parallax shift between cameras is substantially negligible based on the designed operating distance of the camera system to objects 2 and 3 in the scene 1, where larger spacings between the cameras may be tolerated when the designed operating distance is large.

FIG. 2B shows a cross sectional view of two of the cameras or camera modules 10A′ and 10B′ of the camera array 10′ shown in FIG. 2A. As seen in FIG. 2B, each camera or camera module (10A′ and 10B′) includes a corresponding lens, a corresponding image sensor, and may include one or more corresponding filters. For example, in some embodiments, camera 10A′ is a visible light color camera that includes lens 12A′, image sensor 14A′, and color filter 16A′ (e.g., a Bayer filter). In the embodiment shown in FIG. 2B, the filter 16 is located behind the lens 12 (e.g., between the lens 12 and the image sensor 14), but embodiments of the present disclosure are not limited thereto. In some embodiments, the filter 16 is located in front of the lens 12, and in some embodiments, the filter 16 may include multiple separate components, where some components are located in front of the lens and other components are located behind the lens (e.g., a polarizing filter in front of the lens 12 and a color filter behind the lens 12). In some embodiments, camera 10B′ is a polarization camera that includes lens 12B′, image sensor 14B′, and polarizing filter 16B′ (a polarization camera may also include a visible light color filter or other filter for passing a particular portion of the electromagnetic spectrum, such as an infrared filter, ultraviolet filter, and the like). In some embodiments of the present disclosure, the image sensors four cameras 10A′, 10B′, 10C′, and 10D′ are monolithically formed on a same semiconductor die, and the four cameras are located in a same housing with separate apertures for the lenses 12 corresponding to the different image sensors. Similarly, the filters 16 may correspond to different portions of a single physical layer that has different optical filter functions (e.g., different linear polarizing angles or circular polarizers, color filters with corresponding spectral response functions, and the like) in different regions of the layer (corresponding to the different cameras). In some embodiments, a filter 16 of a polarization camera includes a polarization mask 16 similar to the Sony® IMX250MZR sensor, which includes a polarization mosaic aligned with the pixel grid of the image sensor 14 in a manner similar to a red-green-blue (RGB) color filter (e.g., a Bayer filter) of a color camera. In a manner similar to how a color filter mosaic filters incoming light based on wavelength such that each pixel in the image sensor 14 receives light in a particular portion of the spectrum (e.g., red, green, or blue) in accordance with the pattern of color filters of the mosaic, a polarization mask 16 using a polarization mosaic filters light based on linear polarization such that different pixels receive light at different angles of linear polarization (e.g., at 0°, 45°, 90°, and 135°, or at 0°, 60° degrees, and 120°). Accordingly, a camera of the camera array 10′ may use a polarization mask 16 to concurrently or simultaneously capture light at four different linear polarizations.

In some embodiments, a demosaicing process is used to compute separate red, green, and blue channels from the raw data. In some embodiments of the present disclosure, each polarization camera may be used without a color filter or with filters used to transmit or selectively transmit various other portions of the electromagnetic spectrum, such as infrared light.

As noted above, embodiments of the present disclosure relate to multi-modal and/or multi-spectral camera arrays. Accordingly, in various embodiments of the present disclosure, the cameras within a particular camera array include cameras configured to perform imaging in a plurality of different modalities and/or to capture information in a plurality of different spectra.

As one example, in some embodiments, the first camera 10A′ is a visible light camera that is configured to capture color images in a visible portion of the electromagnetic spectrum, such as by including a Bayer color filter 16A′ (and, in some cases, a filter to block infrared light), and the second camera 10B′, third camera 10C′, and fourth camera 10D′ are polarization cameras having different polarization filters, such filters having linear polarization angles of 0°, 60°, and 120°, respectively. The polarizing filters in the optical paths of each of the cameras in the array cause differently polarized light to reach the image sensors of the cameras. The individual polarization cameras in the camera array have optical axes that are substantially perpendicular to one another, are placed adjacent to one another, and have substantially the same field of view, such that the cameras in the camera array capture substantially the same view of a scene as the visible light camera 10A′, but with different polarizations. While the embodiment shown in FIG. 2A includes a 2×2 array of four cameras, three of which are polarization cameras, embodiments of the present disclosure are not limited thereto, and the camera array may more than three polarization cameras, each having a polarizing filter with a different polarization state (e.g., a camera array may have four polarization cameras along with the visible light color camera 10A′, where the polarization cameras may have polarization filters with angles of linear polarization, such as 0°, 45°, 90°, and 135°). In some embodiments, one or more of the cameras may include a circular polarizer.

As another example, one or more of the cameras in the camera array 10′ may operate in other imaging modalities and/or other imaging spectra, such as polarization, near infrared, far infrared, shortwave infrared (SWIR), longwave infrared (LWIR) or thermal, ultraviolet, and the like, by including appropriate filters 16 (e.g., filters that pass light having particular polarizations, near-infrared light, SWIR light, LWIR light, ultraviolet light, and the like) and/or image sensors 14 (e.g., image sensors optimized for particular wavelengths of electromagnetic radiation) for the particular modality and/or portion of the electromagnetic spectrum.

For example, in the embodiment of the camera array 10′ shown in FIG. 2A, four cameras 10A′, 10B′, 10C′, and 10D′ are arranged in a 2×2 grid to form a camera array, where the four cameras have substantially parallel optical axes. In addition, the optical axes of the camera modules of the camera array are arranged close together such that the camera modules capture images from substantially the same viewpoint with respect to the objects in the scene 1. One of skill in the art would understand that the acceptable spacing between the optical axes of the camera modules within an array in order to capture images of the scene from substantially the same viewpoint depends on the working distance to objects 22 in the scene, where longer working distances allow for larger spacing between the optical axes while shorter working distances may require closer or tighter spacing between the optical axes. The four cameras may be controlled together such that they capture images substantially simultaneously. In some embodiments, the four cameras are configured to capture images using the same exposure settings (e.g., same aperture, length of exposure, and gain or “ISO” settings). In some embodiments, the exposure settings for the different cameras can be controlled independently from one another (e.g., different settings for each camera), where the shape estimator 100 jointly or holistically sets the exposure settings for the cameras based on the current conditions of the scene 1 and the characteristics of the imaging modalities and spectral responses of the cameras 10A′, 10B′, 10C′, and 10D′ of the camera array 10′.

In some embodiments, the various individual cameras of the camera array are registered with one another by determining their relative poses (or relative positions and orientations) by capturing multiple images of a calibration target, such as a checkerboard pattern, an ArUco target (see, e.g., Garrido-Jurado, Sergio, et al. “Automatic generation and detection of highly reliable fiducial markers under occlusion.” Pattern Recognition 47.6 (2014): 390-402.) or a ChArUco target (see, e.g., An, Gwon Hwan, et al. “Charuco board-based omnidirectional camera calibration method.” Electronics 7.12 (2018): 421.). In particular, the process of calibrating the targets may include computing intrinsic matrices characterizing the internal parameters of each camera (e.g., matrices characterizing the focal length, image sensor format, and principal point of the camera) and extrinsic matrices characterizing the pose of each camera with respect to world coordinates (e.g., matrices for performing transformations between camera coordinate space and world or scene coordinate space). Different cameras within a camera array may have image sensors with different sensor formats (e.g., aspect ratios) and/or different resolutions without limitation, and the computed intrinsic and extrinsic parameters of the individual cameras enable the shape estimator 100 to map different portions of the different images to a same coordinate space (where possible, such as where the fields of view overlap).

FIG. 2C is a perspective view of a stereo camera array system 10 according to one embodiment of the present disclosure. For some applications, stereo vision techniques are used to capture multiple images of scene from different perspectives. As noted above, in some embodiments of the present disclosure, individual cameras (or camera modules) within a camera array 10′ are placed adjacent to one another such that parallax shifts between the cameras are small or substantially negligible based on the designed operating distance of the camera system to the subjects being imaged (e.g., where the parallax shifts between cameras of a same array are less than a pixel for objects at the operating distance). In addition, as noted above, in some embodiments, differences in the poses of the individual cameras within a camera array 10′ are corrected through image registration based on the calibrations (e.g., computed intrinsic and extrinsic parameters) of the cameras such that the images are aligned to a same coordinate system for the viewpoint of the camera array.

In stereo camera array systems according to some embodiments, the camera arrays are spaced apart from one another such that parallax shifts between the viewpoints corresponding to the camera arrays are detectable for objects in the designed operating distance of the camera system. This enables the distances to various surfaces in a scene (the “depth”) to be detected in accordance with a disparity measure or a magnitude of a parallax shift (e.g., larger parallax shifts in the locations of corresponding portions of the images indicate that those corresponding portions are on surfaces that are closer to the camera system and smaller parallax shifts indicate that the corresponding portions are on surfaces that are farther away from the camera system). These techniques for computing depth based on parallax shifts are sometimes referred to as Depth from Stereo

Accordingly, FIG. 2C depicts a stereo camera array system 10 having a first camera array 10-1′ and a second camera array 10-2′ having substantially parallel optical axes and spaced apart along a baseline 10-B. In the embodiments shown in FIG. 2C, the first camera array 10-1′ includes cameras 10A′, 10B′, 10C′, and 10D′ arranged in a 2×2 array similar to that shown in FIG. 2A and FIG. 2B. Likewise, the second camera array 10-2′ includes cameras 10E′, 10F′, 10G′, and 10H′ arranged in a 2×2 array, and the overall stereo camera array system 10 includes eight individual cameras (e.g., eight separate image sensors behind eight separate lenses). In some embodiments of the present disclosure, corresponding cameras of the camera arrays 10-1′ and 10-2′ are of the same type or, in other words, configured to capture raw frames or images using substantially the same imaging modalities or in substantially the same spectra. In the specific embodiment shown in FIG. 2C, cameras 10A′ and 10E′ may be of a same first type, cameras 10B′ and 10F′ may be of a same second type, cameras 10C′ and 10G′ may be of a same third type, and cameras 10D′ and 10H′ may be of a same fourth type. For example, cameras 10A′ and 10E′ may both have linear polarizing filters at a same angle of 0°, cameras 10B′ and 10F′ may both have linear polarizing filters at a same angle of 45°, cameras 10C′ and 10G′ may both be viewpoint-independent cameras having no polarization filter (NF), such as near-infrared cameras, and cameras 10D′ and 10H′ may both have linear polarizing filters at a same angle of 90°. As another example, cameras 10A′ and 10E′ may both be viewpoint-independent cameras such as visible light cameras without polarization filters, cameras 10B′ and 10F′ may both be thermal cameras, cameras 10C′ and 10G′ may both have polarization masks with a mosaic pattern polarization filters at different angles of polarization (e.g., a repeating pattern with polarization angles of 0°, 45°, 90°, and 135°), and cameras 10D′ and 10H′ may both be thermal (LWIR) cameras.

While some embodiments are described above wherein each array includes cameras of different types in a same arrangement, embodiments of the present disclosure are not limited thereto. For example, in some embodiments, the arrangements of cameras within a camera array are mirrored along an axis perpendicular to the baseline 10-B. For example, cameras 10A′ and 10F′ may be of a same first type, cameras 10B′ and 10E′ may be of a same second type, cameras 10C′ and 10H′ may be of a same third type, and cameras 10D′ and 10G′ may be of a same fourth type.

In a manner similar to that described for calibrating or registering cameras within a camera array, the various polarization camera arrays of a stereo camera array system may also be registered with one another by capturing multiple images of calibration targets and computing intrinsic and extrinsic parameters for the various camera arrays. The camera arrays of a stereo camera array system 10 may be rigidly attached to a common rigid support structure 10-S in order to keep their relative poses substantially fixed (e.g., to reduce the need for recalibration to recompute their extrinsic parameters). The baseline 10-B between camera arrays is configurable in the sense that the distance between the camera arrays may be tailored based on a desired or expected operating distance to objects in a scene—when the operating distance is large, the baseline 10-B or spacing between the camera arrays may be longer, whereas the baseline 10-B or spacing between the camera arrays may be shorter (thereby allowing a more compact stereo camera array system) when the operating distance is smaller.

As noted above with respect to FIG. 1B, a light ray 43 hitting the image sensor 14 of a polarization camera 10 has three measurable components: the intensity of light (intensity image/I), the percentage or proportion of light that is linearly polarized (degree of linear polarization/DOLP/φ, and the direction of that linear polarization (angle of linear polarization/AOLP/ϕ).

Measuring intensity I, DOLP p, and AOLP at each pixel requires 3 or more polarization raw frames of a scene taken behind polarizing filters (or polarizers) at different angles, ϕ_pol (e.g., because there are three unknown values to be determined: intensity I, DOLP ρ, and AOLP ϕ. For example, a polarization camera such as those described above with respect to FIG. 1B captures polarization raw frames with four different polarization angles ϕ_pol, e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees, thereby producing four polarization raw frames I_ϕpol, denoted herein as I₀, I₄₅, I₉₀, and I₁₃₅, and a camera module in accordance with some embodiments of FIGS. 2A, 2B, and 2C may capture polarization raw frames at three different polarization angles ϕ_pol, e.g., 0 degrees, 60 degrees, and 120 degrees, thereby producing three polarization raw frames I_ϕpol denoted herein as I₀, I₆₀, and I₁₂₀.

The relationship between I_ϕpol and intensity I, DOLP ρ, and AOLP ϕ at each pixel can be expressed as:I_ϕpol=I(1+ρ cos(2(ϕ−ϕ_pol)))  (7)

Accordingly, with four different polarization raw frames I_ϕpol (I₀, I₄₅, I₉₀, and I₁₃₅), a system of four equations can be used to solve for the intensity I, DOLP ρ, and AOLP ϕ.

Shape from Polarization (SfP) theory (see, e.g., Gary A Atkinson and Edwin R Hancock. Recovery of surface orientation from diffuse polarization. IEEE transactions on image processing, 15(6):1653-1664, 2006.) states that the relationship between the refractive index (n), azimuth angle (θ_a) and zenith angle (θ_z) of the surface normal of an object and the ϕ and ρ components of the light ray coming from that object follow the following characteristics when diffuse reflection is dominant:

$\begin{array}{cc}\rho =\frac{{\left(n-\frac{1}{n}\right)}^2{\sin }^2\left({\theta }_z\right)}{2+2n^2-{\left(n+\frac{1}{n}\right)}^2{\sin }^2{\theta }_z+4\cos {\theta }_z\sqrt{n^2-{\sin }^2{\theta }_z}}& \left(8\right)\end{array}$ $\begin{array}{cc}\varphi ={\theta }_a& \left(9\right)\end{array}$ and when the specular reflection is dominant:

$\begin{array}{cc}\rho =\frac{2{\sin }^2{\theta }_z\cos {\theta }_z\sqrt{n^2-{\sin }^2{\theta }_z}}{n^2-{\sin }^2{\theta }_z-n^2{\sin }^2{\theta }_z+2s{\mathrm{in}}^4{\theta }_z}& \left(10\right)\end{array}$ $\begin{array}{cc}\varphi ={\theta }_a-\frac{\pi }{2}& \left(11\right)\end{array}$

Note that in both cases p increases exponentially as θ_z, increases and if the refractive index is the same, specular reflection is much more polarized than diffuse reflection.

Accordingly, some aspects of embodiments of the present disclosure relate to applying SfP theory to detect or measure the gradients of surfaces (e.g., the orientation of surfaces or their surface normals or directions perpendicular to the surfaces) based on the raw polarization frames of the objects, as captured by the polarization cameras among the main camera 10 and the support cameras 30. Computing these gradients produces a gradient map (or slope map or surface normals map) identifying the slope of the surface depicted at each pixel in the gradient map. These gradient maps can then be used when estimating the shape and/or pose of the object by supplying these gradient maps or surface normals maps to a trained computer vision model (e.g., a convolutional neural network) and/or by aligning a pre-existing 3-D model (e.g., CAD model) of the object with the measured surface normals (gradients or slopes) of the object in based on the slopes of the surfaces of the 3-D model, as described in more detail below.

One example of an imaging system according to embodiments of the present disclosure includes a stereo pair of 2×2 camera arrays, in an arrangement similar to that shown in FIG. 2C, Each 2×2 camera array includes three color (RGB) cameras with polarization filters at different angles to capture a diverse range of polarization signatures of the scene in the spectral bands (red, green, and blue) and fourth near-IR camera without a polarization filter to capture the scene in the near-IR spectral band. This stereo pair of 2×2 camera arrays may be combined with other cameras located at different viewpoints with respect to the scene, thereby providing a multi-view imaging system. The other cameras may also be similar stereo camera arrays (e.g., similar stereo pairs of 2×2 camera arrays) or monocular camera arrays (e.g., single camera arrays of closely-spaced camera modules), and the camera arrays, in the stereo or monocular case, may have different arrangements and numbers of camera modules in the array (e.g., a 3×2 arrangement of 6 camera modules), and where the camera modules may operate in different modalities (e.g., thermal, ultraviolet, depth from time of flight, polarization, and the like).

Pose Estimation of Known Objects Based on Captured Polarization Information

In some circumstances, the shape estimator 100 has access to 3-D models or computer aided design (CAD) models representing idealized or canonical versions of the objects 22 imaged by the imaging system. These circumstances generally correspond to conditions in which the objects 22 are standardized components that are produced in accordance with those 3-D models, and where each particular real-world instance of the object is substantially identical to each other instance and therefore can be accurately represented by its corresponding known 3-D model. The 3-D model may have been previously generated during the design of the standardized component (e.g., as part of the process of creating the molds) or may be generated through performing a 3-D scan of a part (e.g., using a laser 3-D scanner). Examples of these types of components include manufactured parts, which may be formed through injection molding (in the case of plastics) or casting (in the case of metals). Various surface treatments may be applied to the surfaces of the manufactured parts, which may cause the surfaces of the instances of the objects to have different appearances (e.g., metal parts may be plated, plastic parts may be metalized or coated in metals, various parts may be painted or dyed, and parts may be polished or roughened, and the like).

Examples of techniques for computing estimated poses of known objects for which a 3-D model is available are described in more detail in International Patent Application No. PCT/US21/15926, “Systems and Methods for Object Pose Detection and Measurement,” filed in the United States Patent and Trademark Office on Jan. 29, 2021, U.S. patent application Ser. No. 17/232,084 “Systems and Methods for Six-Degree of Freedom Pose Estimation of Deformable Objects,” filed in the United States Patent and Trademark Office on Apr. 15, 2021, and U.S. patent application Ser. No. 17/314,929, “System and Method for Using Computer Vision to Pick Up Small Objects,” filed in the United States Patent and Trademark Office on May 7, 2021, the entire disclosures of which are incorporated by reference herein.

Generally, some approaches for computing estimated poses of known objects for which a 3-D model is available include determining a class or type of the object (e.g., a known or expected object) and aligning that corresponding 3-D model of the object (e.g., a canonical or ideal version of the object based on known design specifications of the object and/or based on the combination of a collection of samples of the object) with the various views of the object, as captured from different viewpoints around the object. The surface normals of objects in a scene, as computed directly from the polarization information or polarization signatures of surfaces in the scene, provide additional features for properly aligning the 3-D model with the pose of the real-world object in the scene.

FIG. 3 is a flowchart depicting a method for computing six-degree-of-freedom (6-DoF) poses of objects according to some embodiments of the present disclosure.

In operation 310, the shape estimator 100 controls one or more cameras, such as the main camera 10 and the support cameras 30, to capture one or more images of the scene, which may be from multiple viewpoints in the case of multiple cameras. In embodiments using multiple cameras, the cameras are configured to capture images concurrently or substantially simultaneously. Each camera is arranged at a different pose with respect to the scene 1, such that each camera captures scene from its corresponding different viewpoint. Accordingly, the collection of images captured by multiple cameras represent a collection of multi-viewpoint images of the scene 1. (In some embodiments, the images are captured from multiple viewpoints using one or more cameras, such as by moving the one or more cameras between different viewpoints while keeping the scene fixed, and/or rigidly transforming the scene between captures by the one or more cameras.) The one or more images of the scene may be referred to herein as being “consistent” in that they are all pictures of the same consistent scene but providing different views of the scene from different viewpoints and/or different imaging modalities. This consistency between the images of the scene may be achieved by capturing all of the images substantially simultaneously or concurrently or by requiring that none of the objects of interest in the scene that are depicted in the image have moved (e.g., translated or rotated) between in the time between the capture of different images of the scene.

In some circumstances, one or more of the “cameras” are multi-modal cameras that capture multiple images from the same viewpoint, but having different modalities, such as different portions of the electromagnetic spectrum (e.g., red, green and blue portions of the visible light spectrum, near infrared light, far infrared light, ultraviolet light, etc.), different optical filters (e.g., linear polarization filters at different angles and/or circular polarization filters), and combinations thereof. Accordingly, a collection of multi-viewpoint images of a scene does not require that all images be captured from different viewpoints, but only that there are at least two images captured from different viewpoints. Such a collection of multi-viewpoint images therefore may include at least some images that are captured from the same viewpoint.

In the case of a sensing system using multi-viewpoint images or images of a scene from more than one viewpoint, in operation 330, the shape estimator 100 computes object-level correspondences on the multi-viewpoint images of the scene. More specifically, instances of one or more types of objects are identified in the multi-viewpoint images of the scene, and corresponding instances of objects are identified between the multi-viewpoint images. For example, a scene 1 may include two cubes and three spheres, and various of the multi-viewpoint images may depict some or all of these five objects. A process of instance segmentation identifies the pixels in each of the images that depict the five objects, in addition to labeling them separately based on the type or class of object (e.g., a classification as a “sphere” or a “cube”) as well as instance labels (e.g., assigning a unique label to each of the objects, such as numerical labels “1,” “2,” “3,” “4,” and “5”). Computing object-level correspondences between the multi-viewpoint images further relates to computing consistent labels between the different viewpoints (for example, such that the same cube is labeled “1” from each of the viewpoint). Accordingly, the shape estimator 100 generates collections of crops or patches of the multi-viewpoint images of the scene, where each collection of patches depicts the same instance from different viewpoints (cropped to the region containing the object and, in some cases, a small neighborhood or margin around the object).

In the case of a single image depicting a scene from a single viewpoint, in operation 330, the shape estimator 100 may merely compute a segmentation map, which similarly enables the generation of a crop or patch for each object instance detected in the image.

Systems and methods for computing object-level correspondences are described in International Patent Application No. PCT/US21/15926, titled “SYSTEMS AND METHODS FOR POSE DETECTION AND MEASUREMENT,” filed in the United States Patent and Trademark Office on Jan. 29, 2021, which, as noted above, is incorporated by reference herein in its entirety. For the sake of clarity, some techniques for computing object-level correspondences on images are described herein with reference to FIGS. 4A, 4B, and 4C.

In general terms, computing object-level correspondences reduces a search space for conducting image processing tasks such as, for example, pixel-level correspondence. In one embodiment, instance segmentation is performed to identify different instances of objects in images portraying a scene as viewed from different viewpoints, and instance segmentation maps/masks may be generated in response to the instance segmentation operation. The instance segmentation masks may then be employed for computing object level correspondences.

In one embodiment, object level correspondence allows the matching of a first instance of an object appearing in a first image that depicts a view of a scene from a first viewpoint, to a second instance of the same object appearing in a second image that depicts a view of a scene from a second viewpoint. Once object level correspondence is performed, the search space for performing, for example, pixel-level correspondence, may be limited to the regions of the image that correspond to the same object. Reducing the search space in this manner may result in faster processing of pixel-level correspondence and other similar tasks.

FIG. 4A is a flow diagram of a process for object level correspondence according to one embodiment. The process may be implemented by one or more processing circuits or electronic circuits that are components of the shape estimator 100. It should be understood that the sequence of steps of the process is not fixed, but can be modified, changed in order, performed differently, performed sequentially, concurrently, or simultaneously, or altered into any desired sequence, as recognized by a person of skill in the art. The process described with respect to FIG. 4A may be used, in some embodiments of the present disclosure, to compute object level correspondences in operation 330 of FIG. 3, but embodiments of the present disclosure are not limited thereto.

The process starts, and at block 400, the shape estimator 100 receives multi-view images from the main and support cameras 10, 30. A first image captured by one of the cameras may depict one or more objects in a scene from a first viewpoint, and a second image captured by a second camera may depict the one or more objects in the scene from a second viewpoint different from the first viewpoint. The images captured by the cameras may be, for example, polarized images and/or images that have not undergone any polarization filtering.

At block 402 the shape estimator 100 performs instance segmentation and mask generation based on the captured images. In this regard, the shape estimator 100 classifies various regions (e.g. pixels) of an image captured by a particular camera 10, 30 as belonging to particular classes of objects. Each of the different instances of the objects in the image may also be identified, and unique labels be applied to each of the different instances of objects, such as by separately labeling each object in the image with a different identifier.

In one embodiment, segmentation masks delineating the various object instances are also be generated. Each segmentation mask may be a 2-D image having the same dimensions as the input image, where the value of each pixel may correspond to a label (e.g. a particular instance of the object depicted by the pixel). A different segmentation mask may be generated for different images depicting different viewpoints of the objects of interest. For example, a first segmentation mask may be generated to depict object instances in a first image captured by a first camera, and a second segmentation mask may be generated to depict object instances in a second image captured by a second camera. As convolutional neural network such as, for example, Mask R-CNN, may be employed for generating the segmentation masks.

At block 404, the shape estimator 100 engages in object-level correspondence of the objects identified in the segmentation masks. In this regard, the shape estimator may invoke a matching algorithm to identify a segmented instance of a particular object in one image as corresponding (or matching) a segmented instance of the same object in another image. The matching algorithm may be constrained to search for matching object instances along an epipolar line through an object instance in one image to find a corresponding object instance in a different image. In one embodiment, the matching algorithm compares different features of the regions corresponding to the segmented object instances to estimate the object correspondence. The matching of object instances from one image to another may narrow a search space for other image processing tasks such as, for example, performing pixel level correspondence or keypoint correspondence. The search space may be narrowed to the identified regions of the images that are identified as corresponding to the same object.

At block 406, the shape estimator 100 generates an output based on the object-level correspondence. The output may be, for example, a measure of disparity or an estimated depth (e.g., distance from the cameras 10, 30) of the object based on the disparity between corresponding instances as depicted in the various images. In one embodiment, the output is a three-dimensional reconstruction of the configuration of the object and a 6-DoF pose of the object, as described in more detail below with respect to FIG. 3.

FIG. 4B is a block diagram of an architecture for instance segmentation and mask generation of step 402 according to one embodiment. Input images 410 captured by the various cameras 10, 30 are provided to a deep learning network 412 such as, for example, a CNN backbone. In the embodiments where the images include polarized images, the deep learning network may be implemented as a Polarized CNN backbone as described in PCT Patent Application No. PCT/US2020/048604, also filed as U.S. patent application Ser. No. 17/266,046, the content of which is incorporated herein by reference.

In one embodiment, the deep learning network 412 is configured to generate feature maps based on the input images 410, and employ a region proposal network (RPN) to propose regions of interest from the generated feature maps. The proposals by the CNN backbone may be provided to a box head 414 for performing classification and bounding box regression. In one embodiment, the classification outputs a class label 416 for each of the object instances in the input images 410, and the bounding box regression predicts bounding boxes 418 for the classified objects. In one embodiment, a different class label 416 is provided to each instance of an object.

The proposals by the CNN backbone may also be provided to a mask head 420 for generating instance segmentation masks. The mask head 416 may be implemented as a fully convolutional network (FCN). In one embodiment, the mask head 420 is configured to encode a binary mask for each of the object instances in the input images 410.

FIG. 4C is a more detailed flow diagram of a matching algorithm employed at step 404 (FIG. 4A) for identifying object-level correspondence for a particular object instance in a first segmentation mask according to one embodiment. The process may repeat for all object instance identified in the first segmentation mask. The sequence of steps of the process of FIG. 4C is not fixed, but can be modified, changed in order, performed differently, performed sequentially, concurrently, or simultaneously, or altered into any desired sequence, as recognized by a person of skill in the art.

At block 430, the matching algorithm identifies features of a first object instance in a first segmentation mask. The identified features for the first object instance may include a shape of the region of the object instance, a feature vector in the region, and/or keypoint predictions in the region. The shape of the region for the first object instance may be represented via a set of points sampled along the contours of the region. Where a feature vector in the region is used as the feature descriptor, the feature vector may be an average deep learning feature vector extracted via a convolutional neural network.

At block 432, the matching algorithm identifies an epipolar line through the first object instance in the first segmentation mask.

At block 434, the matching algorithm identifies one or more second object instances in a second segmentation mask that may correspond to the first object instance. A search for the second object instances may be constrained to the epipolar line between the first segmentation map and the second segmentation map that runs through the first object instance. In one embodiment, the matching algorithm searches approximately along the identified epiploar line to identify object instances in the second segmentation mask having a same class identifier as the first object instance. For example, if the first object instance belongs to a “dog” class, the matching algorithm evaluates object instances in the second segmentation mask that also belong to the “dog” class, and ignores objects that belong to a different class (e.g., a “cat” class).

At block 436, the matching algorithm identifies the features of the second object instances that belong the same class. As with the first object instance, the features of a particular second object instance may include a shape of the region of the second object instance, a feature vector representing the region, and/or keypoint predictions in the region.

At block 438, the matching algorithm compares the features of the first object instance to the features of second object instances for determining a match. In one embodiment, the matching algorithm identifies a fit between the features of the first object instance and features of the second object instances for selecting a best fit. In one embodiment, the best fit may be identified via a matching function such as the Hungarian matching function. In one embodiment, the features of the object instances are represented as probability distributions, and the matching function attempts to find a match of the probability distributions that minimizes a Kullback-Leibler (KL) divergence.

At block 440, a determination is made as to whether a match has been found. If the answer is YES, an output is generated at block 442. The output may include, for example, information (e.g. object ID) of the second object instance that matched the first object instance.

If the answer is NO, an output may be generate indicating a match failure at block 444.

Accordingly, object level correspondences can be computed from the multi-viewpoint images. These object level correspondences may be used to extract corresponding crops or patches from the multi-viewpoint images, where each of these crops or patches depicts a single instance of an object, and collections of corresponding crops or patches depict the same instance of an object from multiple viewpoints.

In operation 350, the shape estimator 100 loads a 3-D model of the object based on the detected object type one or more object detected in the scene (e.g., for each detected instance of a type of object). For example, in a circumstance where the collection of objects 22 includes a mixture of different types of flexible printed circuit boards, the process of computing object-level correspondences assigns both an instance identifier and a type (or classification) to each detected instance of a flexible printed circuit board (e.g., which of the different types of printed circuit boards). Therefore, a 3-D model of the object may then be loaded from a library based on the detected object type.

In operation 370, the shape estimator 100 aligns the corresponding 3-D model to the appearances of the object to be consistent with the appearance of the object as seen from the one or more viewpoints. In the case of deformable objects, the alignment process in operation 370 may also include deforming the 3-D model to match the estimated configuration of the actual object in the scene. This alignment of the 3-D model provides the 6-DoF pose of the object in a global coordinate system (e.g., a coordinate system based on the main camera 10 or based on the robot controller 28). Details of aspects of the present disclosure for performing the alignment of a 3-D model with the appearance of an object will be described in more detail below.

Generally, the methods described herein will make use of a 3-D model or computer-aided-design (CAD) model C of the object (e.g., as loaded in operation 350) and observed two-dimensional (2-D) image data I of the object (e.g., as captured by the cameras in operation 310 and with object-level corresponding patches of the images extracted therefrom in operation 330). In some embodiments, the output of the 6-DoF pose estimation technique (computed by the shape estimator 100) includes a mesh M and its 6-DoF pose in a global coordinate system (e.g., 3 dimensional translational and rotational coordinates in a coordinate system oriented with respect to a main camera 10) for each of the detected objects in the scene.

To align a 3-D model with the observed 6-DoF pose of an object in a scene, embodiments of the present disclosure generally attempt to find a pose of the 3-D model that causes its appearance, from one or more virtual cameras, to be consistent with the one or more observed images of the object captured by the cameras 10, 30. Generally, these approaches include detecting keypoints in the object level patches of the images, and transforming the pose of the 3-D model such that the locations of the keypoints in the 3-D model are consistent with the locations of the keypoints in the observed images. In circumstances where the images of the scene also include one or more depth maps, the 3-D model may also be aligned with the depth maps through a 3-D model alignment algorithm such as iterative closest point (ICP).

In circumstances where the images include surface normals maps (e.g., computed from polarization signatures of the object based on shape-from-polarization, as described above), the pose of the 3-D model is further aligned with the observed surface normals. For example, in some embodiments, the correspondences between the locations of keypoints in the observed images and locations on the 3-D model are identified, and the directions of the surface normals at corresponding portions of surface normals map are compared against corresponding directions of the surface normals on the 3-D model to compute an error that is used as part of an error function for aligning the pose of 3-D model with the actual pose of the observed object. In some embodiments, the correspondences are computed based on identifying matching keypoints using a keypoint detector (e.g., a classical keypoint detector or a trained neural network based keypoint detector), (e.g., updating an estimated pose of the 3-D model to minimize differences or errors between the locations of the keypoints in the observed images and the locations of the keypoints in 3-D model of the object, a render-and-compare approach (e.g., by using a differentiable rendering engine, where the differences or errors between detected keypoints and locations of keypoints in renderings of the 3-D model are propagated backward through the differentiable rendering engine to update the pose, see, e.g., Labbé, Yann, et al. “CosyPose: Consistent multi-view multi-object 6D pose estimation.” European Conference on Computer Vision. Springer, Cham, 2020.), or dense correspondences between surfaces of 3-D models and surfaces of objects and may be computed as described in more detail below with respect to FIG. 5.

Some approaches to aligning 3-D model to their appearances in images relate to computing dense correspondences between surfaces of the object depicted in the one or more images of the scene and surfaces of the 3-D model by rendering images of the 3-D model in an initial (or current) estimated pose.

FIG. 5 is a flowchart depicting a method 500 for computing a pose of an object based on dense correspondences according to some embodiments of the present disclosure. For the sake of clarity, embodiments of the present disclosure will be described with respect to the estimation of the pose of one object in the scene. However, embodiments of the present disclosure are not limited thereto and include embodiments wherein the pose estimator 100 estimates the poses of multiple objects in the scene as depicted in the one or more images captured in operation 310 (e.g., where the poses of the multiple objects may be estimated in parallel or jointly in a combined process).

In operation 510, the pose estimator 100 computes an initial pose estimate of an object based on one or more images of the object, such as the image patches extracted in operation 330. The pose estimator 100 may also receive one or more 3-D models corresponding to the detected objects (e.g., as loaded in operation 350) where the 3-D model is posed (e.g., translated and rotated) based on the initial pose estimate. In some embodiments, the initial pose estimate is computed based on detecting keypoints in the one or more images of the object and using a Perspective-n-Point algorithm to match the detected keypoints with corresponding known locations of keypoints in the 3-D model. See, e.g., Zhao, Wanqing, et al. “Learning deep network for detecting 3D object keypoints and 6D poses.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. and Lepetit, Vincent, Francesc Moreno-Noguer, and Pascal Fua. “EPnP: An accurate O(n) solution to the PnP problem.” International Journal of Computer Vision 81.2 (2009): 155. The keypoints may be detected using, for example, a classical keypoint detector (e.g., scale-invariant feature transform (SIFT), speeded up robust features (SURF), gradient location and orientation histogram (GLOH), histogram of oriented gradients (HOG), basis coefficients, Haar wavelet coefficients, and the like.) or a trained deep learning keypoint detector such as a trained convolutional neural network using HRNet (Wang, Jingdong, et al. “Deep high-resolution representation learning for visual recognition.” IEEE transactions on pattern analysis and machine intelligence (2020).) with a differential spatial to numerical (DSNT) layer and Blind Perspective-n-Point (Campbell, Dylan, Liu, and Stephen Gould. “Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization.” European Conference on Computer Vision. Springer, Cham, 2020.).

As another example, the initial pose estimate may be computed by capturing a depth image or depth map of the object (e.g., using a stereo depth camera or time of flight depth camera) and applying an iterative closest point (ICP) algorithm or a point pair feature matching algorithm (see, e.g., Drost, Bertram, et al. “Model globally, match locally: Efficient and robust 3D object recognition.” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2010.) to align the 3-D model to the shape of the object as it appears in the depth image. In some embodiments, the initial pose estimate is computed directly from a trained network (see, e.g., Xiang, Yu, et al. “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes.” arXiv preprint arXiv:1711.00199 (2017).) and/or approaches such as a dense pose object detector (Zakharov, Sergey, Ivan Shugurov, and Slobodan Ilic. “DPOD: 6D Pose Object Detector and Refiner.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, 2019.)

FIG. 6 is a schematic depiction of a 3-D model, depicted in shaded form, posed in accordance with an initial pose estimate and overlaid onto an observed image of a scene, depicted in line drawing form. As shown in FIG. 6 these is an error between the observed object 602 and the rendering of the 3-D model 604 as posed based on the initial pose estimate, both in the form of rotation error and translation error. Accordingly, aspects of embodiments of the present disclosure relate to refining this initial pose estimate (whether performed using keypoint detection and a PnP algorithm or using a depth image and an ICP algorithm as discussed above, or through other techniques) as described in more detail below.

FIG. 7A is a block diagram depicting a pipeline 700 for refining an initial pose estimate using dense correspondences according to one embodiment of the present disclosure. In various embodiments, the pipeline 700 is implemented in whole or in part by the pose estimator 100 to compute refined pose estimates, or feature vectors in other representation spaces representing the location of the object, based on input images of the object.

Referring back to FIG. 5 and to FIG. 7A, in operation 530, the pose estimator 100 uses a renderer 710 (or rendering engine) to render an image 731 (e.g., a 2-D image) of the 3-D model 711 in its initial pose 712 from the viewpoint of a camera (e.g., as specified by extrinsic camera parameters) that captured an image of the object in the scene. In embodiments in which multiple consistent images of the object were captured from multiple viewpoints, the pose estimator 100 renders a separate image of the 3-D model in its initial estimated pose in the scene observed by the cameras from each of the separate viewpoints with respect to the object in the scene. The rendering may also be performed in accordance with camera intrinsic parameters (e.g., accounting for field of view and lens distortions of the camera or cameras used to capture the observed images of the object in the scene).

In some embodiments of the present disclosure, the rendered image of the object is a rendered surface normals map, where each pixel or point in the rendered surface normals map is a vector indicating the direction of the surface of the 3-D model depicted at that pixel or point (e.g., a vector perpendicular to the surface of the object at that pixel or point). In some cases, the normal vector at each pixel is encoded in the color channels of an image (e.g., in red, green, and blue color channels). In some embodiments, the pose estimator 100 renders the rendered surface normals map by computing a depth map from the perspective or viewpoint of the observing camera used to capture the observed image (e.g., using the Moller-Trumbore ray-triangle intersection algorithm as described in Möller, Tomas, and Ben Trum bore. “Fast, minimum storage ray-triangle intersection.” Journal of graphics tools 2.1 (1997): 21-28.). According to these embodiments, the depth map of the object is converted to a point cloud, and a rendered surface normals map is computed from the point map (e.g., by computing the slope between neighboring or adjacent points of the point cloud).

In some embodiments of the present disclosure, the pose estimator 100 renders the rendered surface normals map directly from 3-D model with a virtual camera placed at the perspective or viewpoint of the observing camera. This direct rendering may be performed by tracing rays directly from the virtual camera into a virtual scene containing the 3-D model in its initial estimated pose and computing the surface normal of the first surface that each ray intersects with (in particular, the surfaces of the 3-D model in the initial estimated pose that the rays intersect with).

While the rendered image 731 in the embodiments described above include one or more rendered surface normals maps, embodiments of the present disclosure are not limited thereto and the renderer may be configured to generate different types of rendered 2-D images such as color (e.g., red, green, blue) images, monochrome images, and the like.

In operation 570, the pose estimator 100 computes dense image-to-object correspondences between the one or more images of the object and the 3-D model of the object. For example, the rendered image 731 of the object in the scene based on the initial estimated pose and observed image 732 of the object in the same scene (or multiple rendered images 731 and multiple observed images 732 from different viewpoints) are supplied to correspondence calculator 730, which computes dense correspondence features between the rendered image 731 and the observed image 732 (or the rendered images 731 and the corresponding observed images 732 of the object in the scene).

In various embodiments, the correspondence calculator 730 may use different techniques to compute dense correspondence features between the rendered image 731 and the observed image 732. In some embodiments, a disparity neural network is used to detect correspondences (see, e.g., Xu, Haofei, and Juyong Zhang. “AANet: Adaptive aggregation network for efficient stereo matching.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.), where the disparity neural network is modified to match pixels along the y-axis of the images (e.g., perpendicular to the usual direction of identifying correspondences by a disparity neural network) in addition to along the x-axis of the input images (as traditional, where the input images are rectified to extend along the x-axis between stereo pairs of images), where the modification may include flattening the output of the neural network before supplying the output to the loss function used to train the disparity neural network, such that the loss function accounts identifies and detects disparities along both the x-axis and the y-axis. In some embodiments, an optical flow neural network is trained and/or retrained to operate on the given types of input data (e.g., observed surface normals maps and observed images), where examples of optical flow neural networks are described in Dosovitskiy, Alexey, et al. “FlowNet: Learning optical flow with convolutional networks.” Proceedings of the IEEE international conference on computer vision. 2015. IIg, Eddy, et al. “FlowNet 2.0: Evolution of optical flow estimation with deep networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. and Trabelsi, Ameni, et al. “A Pose Proposal and Refinement Network for Better 6D Object Pose Estimation.” Proceedings of the IEEE/C VF Winter Conference on Applications of Computer Vision. 2021. In some embodiments, classical techniques for computing dense correspondences are be used, such as classical algorithms for computing optical flow (see, e.g., Horn and Schunck, referenced above) or classical techniques for computing disparity (e.g., block matching, but applied along both the x-axis and y-axis). Other embodiments of the present disclosure include modifications and/or retraining of existing neural network backbones to take two inputs (e.g., the observed image and the rendered image) to compute correspondences.

The observed image or observed images 732 supplied as input to the correspondence calculator 730 may be the same images that were used to compute the initial pose estimate or may be different images, such as images from different viewpoints from those used to compute the initial pose estimate, images captured in different modalities (e.g., polarization and/or different spectra), or images or feature maps computed based on captured or observed images (e.g., observed features in polarization representation spaces or observed surface normals computed from polarization features using shape-from-polarization techniques). Examples of types of images include color images (e.g., red, green, blue images) captured by color cameras, monochrome images (e.g., in the visible light, infrared, or ultraviolet portions of the spectrum), polarization raw frames (e.g., color or monochrome images captured through a polarization filter), polarization features in polarization representation spaces (e.g., angle of linear polarization (AOLP) and degree of linear polarization (DOLP)). As discussed in more detail above, shape from polarization (SfP) provides techniques for computing observed surface normals maps from captured or observed polarization raw frames.

Accordingly, the correspondence calculator 730 computes dense correspondences between the rendered image 731 and the observed image 732.

Through the rendering process, the pose estimator 100 also stores information associated with the rendered image 731 regarding the point in the 3-D model that is represented by each pixel in the rendered image. For example, when rendering the image using a ray tracing technique, each pixel of the rendered image corresponds to a location on the surface of the 3-D model (e.g., in uv coordinate space representing points on the surface of the 3-D model) as defined by a ray connecting the camera origin, the pixel, and the location on the surface of the 3-D model, as modified by any virtual optics system (e.g., as defined by camera intrinsic parameters). As such, the pose estimator 100 stores 2-D to 3-D correspondences between the 2-D rendered image 731 and the 3-D model in its initial pose.

Therefore, the correspondence calculator 730 further computes dense image-to-object correspondences 740 that maps pixels in the observed image 732 to locations on the surface of the 3-D model 711. In more detail, as shown in FIG. 7B, the optical flow features computed by the correspondence calculator 730 provide a mapping from pixels in the observed image 732 to pixels in the rendered image 731 and the 2-D to 3-D mapping information from the rendering process provides mappings from pixels in the rendered image 731 to locations on the surface of the 3-D model 711. As a result, the dense image-to-object correspondences 740 provide 2-D to 3-D correspondences between every visible pixel in the observed image 732 and the predicted point it represents on the 3-D model 711 of the object.

In operation 590, the pose estimator 100 updates the estimated pose based on the dense image-to-object correspondences. For example, as shown in FIG. 7A, the dense image-to-object correspondences may be supplied to a Perspective-n-Point (PnP) algorithm to compute a refined pose estimate. In some embodiments, the PnP algorithm estimates the refined pose P by finding the pose P that minimizes the error function below:

$\underset{P}{\arg \min }\sum _{x\in X}\mathrm{KPf}\left(x\right)-x$ where K is the camera intrinsic matrix of the camera used to capture the observed image of the object, P is a pose matrix representing the transformation between the object and the camera, f: ²→³ is the dense image-to-object correspondences described above (computed in operation 570) mapping from pixel coordinates in the observed image to 3-D coordinates on the surface of the 3-D model, and X is the domain of f (e.g., across all of the pixels in the observed image of the object).

Because the correspondence calculator 730 computes a large number of correspondences (e.g., dense correspondences) between the image and the 3-D model of the object, these correspondences can also be used to estimate the configuration of the deformable object using a PnP algorithm, thereby enabling the measurement of the configuration of deformable objects (e.g., bags holding loose items such as food, clothes, flexible printed circuit boards, and the like) by deforming the 3-D model to match the configuration of the object. In some embodiments, the deformation of the 3-D model to match the configuration of the deformable object in the images can be computed for every pixel coordinate x∈X (where X represents the collection of all pixels in the observed images) as:{Pf(x)−proj_L(x)(Pf(x))|x∈X}where L(x) represents a line of a projection of point x from the camera, P is a pose matrix representing the transformation between the object and the camera, f: ² →³ is the dense image-to-object correspondences described above (computed in operation 570) mapping from pixel coordinates in the observed image to 3-D coordinates on the surface of the 3-D model, proj_L(x) (Pf(x)) is the estimated depth of the object coordinate seen at point x from the camera along line L(x), and X is the domain off (e.g., across all of the pixels in the observed image of the object). Accordingly, the above expression provides one estimate of the deformation of the object, e.g., the difference between the predicted location based on the current pose P and a 3-D model of the object (as represented by the term Pf(x)) and the actual observed location of the corresponding point in the observed image, as represented by the term proj_L(x) (Pf(x)), where the difference represents the change in 3-D coordinates to be applied to make the shape of the 3-D model match up with the actual deformed shape or configuration of the observed object.

In some embodiments where a depth map D of the scene is available (e.g., by capturing a depth map of the scene using a depth camera such as a stereo camera) among the one or more observed images 732, the depth map is used to convert the pixel coordinates x to 3-D coordinates D(x) and therefore the deformation would be computed for each pixel x as:{Pf(x)−D(x)|x∈X}Accordingly, the above expression provides one estimate of the deformation of the object, e.g., the difference between the predicted location based on the current pose P and a 3-D model of the object (as represented by the term Pf(x)) and the actual observed location of the corresponding point in the observed depth image D(x), where the difference represents the change in 3-D coordinates to be applied to make the shape of the 3-D model match up with the actual deformed shape or configuration of the observed object.

While FIG. 5 shows an embodiment where an updated pose of the 3-D model is computed once, in some embodiments the pose is iteratively refined by supplying the pose computed in operation 590 as the initial pose of the next iteration in operation 530 in order to further refine the estimated pose of the object for consistency with the observed image of the object.

In addition, while FIG. 5 depicts a circumstance in which the observed image of the object is captured from a single viewpoint, embodiments of the present disclosure are not limited thereto and may be applied in a multi-view environment where multiple cameras (e.g., a main camera 10 and support cameras 30) capture observed images of the object from multiple different viewpoints. In such embodiments, the multiple views (e.g., N different views) may be jointly used to compute a pose estimate that minimizes a combined error metric across the multiple views (e.g., errors computed by comparing the locations of keypoints in the observed images from each viewpoint with renderings from each viewpoint).

Generating Datasets of Images of Known Objects and Corresponding Shape Estimates

As noted above, in some embodiments, the shape estimator 100 includes a renderer 150 such as a 3-D rendering engine that is configured to compute shape estimates of the objects detected in the scene based on estimated poses of those objects.

FIG. 8 is a flowchart depicting a method for generating datasets including images of known objects and corresponding shape estimates according to one embodiment of the present disclosure.

In operation 810, the shape estimator controls one or more cameras to capture one or more images of a scene containing known objects. The images may be captured in accordance as described above with respect to operation 310 of FIG. 3, such as by controlling a main camera 10 and support cameras 30 to capture consistent images of a scene from one or more viewpoints. The images may include images in different modalities, such as images covering different parts of the electromagnetic spectrum (e.g., visible light of different colors, near infrared, thermal, ultraviolet, and the like), depth maps (e.g., captured using depth from active or passive stereo), polarization raw frames captured by cameras with different polarization filters (e.g., circular polarization filters or linear polarization filters at different angles), and combinations thereof.

In some embodiments, the images may also include polarization signatures or polarization signature maps computed from the raw images from the cameras, including Stokes vectors, degree of linear polarization (DOLP), and angle of linear polarization (AOLP) (for cases where there are three or more polarization raw frames captured with different polarization angles from the same viewpoint). In some cases, the images also include physics-based surface normals maps (e.g., N_x, N_y, and N_z for each pixel), where these normals maps may be computed from the polarization signatures based on shape from polarization techniques, as described above.

In the case of a multi-viewpoint system, such as where multiple ones of the main camera 10 and the support cameras 30 include multi-modal camera systems (e.g., monocular multi-modal camera arrays and stereo multi-modal camera arrays), the above values can be estimated for multiple viewpoints. While the 3-D depth or 3-D coordinates of each point of the objects visible in the scene will be consistent (within expected noise tolerances) in the depth maps captured across the viewpoints of the multiple cameras, the DOLP and AOLP will vary depending on the viewpoint as well as the color of the object and as a result each of the normal maps will be different. For each viewpoint and each color channel, there is a corresponding set of surface normals N estimated from the physics of polarization by applying the Fresnel equation which will vary depending on the viewpoint due to a number of factors that include: material reflectivity or “albedo,” wavelength, and specular reflections/viewing direction.

Regarding material reflectivity or “albedo,” when the albedo of the material is low it has a significant impact on polarization. Umov's law which states that the albedo and the degree of polarization are inversely proportional to one another. (For example, low albedo materials have a very high degree of polarization, while high albedo materials have a low degree of polarization.) In these cases of low albedo materials, it is likely that the surface normals estimated from different viewpoints are substantially similar (after accounting for the rigid body transformation between the two viewpoints), whereas the estimated surface normals based on physics may be very different for high albedo materials.

Regarding wavelength, the albedo is wavelength dependent for a whole range of colors other than pure black and pure white (e.g., the albedo of black car and a white car are spectrally invariant over the visible wavelength range). As a result, the degree of polarization is stronger for certain colors and its corresponding signal-to-noise ratio (SNR) is higher. Therefore, the surface normals estimated from certain, low albedo wavelength channels will be more accurate than those from other channels with higher albedo.

Regarding specular reflections and viewing direction, specular reflections change with viewing direction as well as illumination direction. The brightness variations that result from specular reflection (and not material geometry) are sometimes referred to as “texture-copy artifacts.” In such situations, having a substantially different viewpoint will result in a change in brightness to one which is more consistent with the material and geometry of the object. In that case, the surface normals estimated from viewpoints where texture-copy artifacts are not visible on the surface of the object are likely to be more accurate than those from other viewpoints in which texture-copy artifacts do appear on the surface of the object.

In some embodiments, the observed images take the form of:

• • I_r1, I_g1, I_b1, N_r1d, N_r1s1, N_g1d, N_g1s1, N_g1s2, N_b1d, N_b1s1, N_b1s2 I_r2, I_g2, I_b2, N_r2d, N_r1s1, N_g2d, N_g2s1, N_g2s2, N_b2d, N_b2s1, N_b2s2 . . . I_rn, I_gn, I_bn, N_rnd, N_r1n1, N_gnd, N_gns1, N_gns2, N_bnd, N_bns1, N_bns2 where I_ri, I_gi, I_bi, represent intensity images from viewpoint i among viewpoints 1 through n in r, g, and b spectral channels, N_rid, N_ris1, N_ris2, N_gid, N_gis1, N_gis2, N_btid, N_btis1. N_bis2 are the surface normals estimated from diffuse and specular reflection models on the red, green, and blue spectral channel images using a physics-based approach that leverages the Fresnel equations (e.g., based on shape from polarization).

According to some embodiments, pose estimation and data generation systems of the present disclosure are deployed in factory conditions where the illumination conditions are not always known or may not be uniform (e.g., periodically changing illumination due to moving machinery may change the illumination conditions from one image to the next). Accordingly, applying photometric constraints is challenging given the varying illumination conditions. Imaging systems according to some embodiments of the present disclosure capture multi-channel polarization information to provide additional constraints for disambiguating (or reducing ambiguity) in the surface normals computed based on a computer vision model trained based on datasets generated in accordance with embodiments of the present disclosure. For example, as discussed in more detail below, the polarization images as well as the corresponding estimated normal maps may be supplied to train a computer vision model (e.g., a deep learning network) to choose how to combine these inputs effectively to form the desired output, such as shape estimates of objects depicted in the images such as depth maps and surface normals maps that have low noise compared to comparative approaches (e.g., using depth from stereo, depth from time of flight, and surface normals from shape from polarization using the Fresnel equations).

In operation 830, the shape estimator 100 computes pose estimates of the known objects depicted in the scene. These pose estimates may be computed from the one or more images based on the pose estimation techniques such as those described above with respect to FIGS. 3, 4A, 4B, 4C, 5, 6, 7A, and 7B. In some example embodiments, the multi-view light field capture of the scene using a main camera 10 and multiple support cameras 30 is used to estimate depths of surfaces of objects using multi-view stereo correspondence, correlates the depths of these surfaces with the detected keypoints in the images and keypoints in the ground truth 3-D CAD models of the objects in the images, and jointly optimizes the depth errors over the multiple viewpoints and multiple imaging modalities to provide 6DoF pose estimations.

In some embodiments, the joint optimization of depth errors includes computing surface shape estimates based on the input images using a shape estimation neural network, and including the differences between the shape estimates and the rendered shapes (e.g., comparing the estimated depth maps to rendered depth maps and/or comparing the estimated surface normals maps of the 3-D model in an initial estimated pose to the rendered surface normals maps of the 3-D model in the initial estimated pose to update the estimated pose of the 3-D model). This shape estimation neural network may be trained based on existing training data mapping input images and image signatures (e.g., polarization signatures) to ground truth smooth shapes. In some circumstances the differences or errors computed in accordance with the different factors are separately weighted, such as based on relative confidences output by the different factors. For example, the shape estimation neural network may include a confidence score as part of its output and, as another example, depth maps and surface normals maps computed using shape from polarization approaches may be weighted based on the level of noise present in the underlying images.

However, embodiments of the present disclosure are not limited thereto, and the poses of the objects may be estimated using different techniques, as discussed above.

In operation 850, the renderer 150 renders shape estimates of the objects in the scene based on 3-D models of the objects posed based on the estimated poses. These rendered shape estimates represent the “ground truth” or desired output labels associated with the captured images. Using the pose estimates computed in operation 830 and ground truth, accurate 3-D CAD models of the objects, the shape estimator 100 infers the shape of the objects, including surface normal maps, accurately enough to have them represent ground truth for the purpose of populating a dataset for training computer vision models, such as by adding a new data point to a collection of data points of the dataset in operation 870.

These rendered shape estimates may include images of a virtual scene with one or more 3-D models posed in accordance with the estimated poses of the corresponding objects depicted in the scene, and the images are rendered from the perspective of virtual cameras having intrinsic and extrinsic camera parameters matching those of the main camera 10 and/or support cameras 30, such that each rendered view corresponds to a view from the observed images 18. For example, rendering color images of a scene may provide estimates regarding the outline or silhouette of the object as viewed from a particular camera. As another example, rendering a depth map based on the 3-D model can provide a high resolution depth map or point cloud of the shape of the objects in the scene with substantially no noise compared to point clouds or depth maps generated from depth camera systems. As a third example, rendering a surface normals map of the surface normals directions of the surfaces of the 3-D model produces a higher resolution surface normals map with substantially no noise compared to surface normals maps computed directly from depth maps from depth camera systems or computed from polarization raw frames based on shape from polarization techniques. In various embodiments, these surface normals maps may be rendered by directly detecting the surface normals (e.g., the angles or slopes or orientations) of the surfaces of the posed 3-D model, or may be computed from the high resolution depth map rendered from the 3-D model (e.g., by computing the gradient between adjacent pixels of the depth map or adjacent points of the point cloud).

In operation 870, the shape estimator 100 adds a data point to a collection of data points, where the added data point includes the one or more observed images of the scene (as captured by the one or more cameras) and the corresponding shape estimates from the same viewpoint with respect to the object (e.g., as generated by the renderer). In some cases, each data point includes one or more images from a given viewpoint and one or more shape estimates (e.g., rendered images) rendered from the corresponding virtual viewpoint. In some embodiments, the shape estimator 100 determines whether or not to add a particular data point to the collection of data points in operation 870 based on whether the robotic arm 24 was able to pick an object depicted in the images, or otherwise perform a particular task, based on the estimated 6-DoF pose of the object as computed from the observed images of the scene associated with the data point. In other words, the ability of the robotic controller to control the robotic arm to pick the object may be included as a factor in validating the rendered “ground truth” surface normals of the object depicted in the images and therefore in determining whether or not the data point should be included in the generated dataset.

These observed images are paired with their corresponding “ground truth” shape signatures generated by the renderer from the posed 3-D model. For example, ground truth surface normals maps generated by the renderer may be substantially similar to the surface normals N_rid, N_ris1, N_ris2, N_gid, N_gis1, N_gis2, N_bid, N_bis1, N_bis2 estimated from the captured polarization images, but will generally be smoother (e.g., have less noise or substantially no noise) than the observed images because the ground truth surface normals maps were generated from a virtual rendering environment in which the directions of the surface normals of the posed 3-D model are known.

As such, aspects of embodiments of the present disclosure provide systems and methods for generating datasets that include observed images of real-world objects and corresponding ground truth images or signatures representing the shapes of those real-world objects, where the ground truth images or signatures may include high resolution, low noise or substantially noise-free depth maps and surface normals maps of the objects.

As described above, the dataset may be generated, in part, by shape estimators 100 operating as pose estimators for estimating the poses of known objects in a manufacturing environment, where the computed poses are supplied to a controller 28 for controlling a robotic arm 24 to pick the objects based on their computed poses. The collection of data as part of an existing pose estimation process in accordance with some embodiments of the present disclosure generates a large number of data points relating to known objects under a variety of different conditions (e.g., appearing in different orientations, under varying lighting conditions, interacting with various other objects, and the like). Some aspects of embodiments of the present disclosure relate to aggregating data points collected from diverse environments (e.g., different shape estimators operating on different logistics facilities or manufacturing lines that are configured to compute the poses and/or shapes of different objects, such as different manufacturing lines that manufacture different products from different components, and where the different facilities may be operated by different entities). Accordingly, embodiments of the present disclosure provide systems and methods for generating large and diverse datasets for training computer vision models to perform computer vision tasks such as shape estimation, such as the slope or surface normals map of an object.

Computer Vision Models Trained Based on Datasets Including Slope Data

Datasets generated in accordance with embodiments of the present disclosure may be applied to train computer vision models such as deep neural networks (e.g., convolutional neural networks) to compute images or signatures representing the shapes of objects that are depicted in one or more given input images. Considering the arrangement shown in FIG. 1A, a main camera 10 and support cameras 30 may capture images of a scene, such as polarization raw frames in different portions of the visible spectrum (e.g., red, green, and blue color channels of color images), infrared images, depth maps from stereo, and the like. In addition, shape from polarization may be used to compute surface normals maps from polarization signatures (e.g., AOLP and DOLP) based on the polarization raw frames in accordance with the Fresnel equations.

In more detail, these aspects of embodiments of the present disclosure relate to training a computer vision model such as a neural network to implement shape from polarization and/or multi-view/multi-spectral stereo given a set of input images. In particular, neural networks are capable of performing non-parametric functional approximations and therefore can be trained to compute the desired mapping between input space (e.g., captured multi-modal and multi-view images and/or signatures such as polarization signatures, noisy depth maps, and noisy surface normals maps) and the corresponding shape of the depicted objects (e.g., surface orientation in the form of surface normals maps and/or depth maps).

FIG. 9 is a schematic block diagram depicting training a computer vision model using a dataset according to some embodiments of the present disclosure. As shown in FIG. 9, a training dataset 910 including captured images and corresponding clean ground-truth shape data (e.g., corresponding surface normals maps) is supplied with along with an untrained computer vision model 920 to a model training system 930 to compute a trained computer vision model 940 based on the training dataset 910. In particular, the training process may compute a plurality of parameters that configure the computer vision model to perform particular tasks. As a specific example, in the case of deep convolutional neural network, the trained parameters may include weights and biases of connections between the neurons in various layers of the deep convolutional neural network. The untrained computer vision model 920 may have initial parameters that are set randomly or may be a pre-trained network that may have previously been trained to perform a different task or trained based on a different training dataset. Captured images 950 of a scene may then be supplied to the trained computer vision model 940 to compute shape estimates by the computer vision model 960. In comparison, a comparative shape estimator 970 that computed the shapes of objects directly from the images (e.g., without using a trained computer vision model) may use techniques such as depth from disparity and/or shape from polarization to compute shape estimates directly from the images 980 (these may be referred to as being computed using “classical” techniques as opposed to techniques based on machine learning or statistical learning). However, these classically computed shape estimates may exhibit noisy or inconsistent estimates.

As such, a neural network trained according to embodiments of the present disclosure would disambiguate between the different noisy or inconsistent surface normals N computed from the different views, modalities, and spectral information captured (e.g., where the surface normals maps N may differ in accordance with the noise or variability of the polarization signature due to differences in albedo, wavelength, and texture copy artifacts due to viewing direction, as well as other noise in ambiguity in the image capture process). Comparing the shape estimates from the model 960 against the shape estimates computed directly from the images 980 in a comparison module 990 show that a properly-trained computer vision model 940 produces smoother and more accurate shape estimates than comparative techniques.

In addition, given a dataset depicting a sufficiently diverse set of objects (e.g., generated based on images of known objects and their corresponding 3-D models), in some embodiments, the trained model is generalized to generate accurate and low-noise estimates of the shapes of arbitrary objects (e.g., objects that may not depicted in the training dataset), thereby enabling the estimation of unknown or novel objects (e.g., objects for which corresponding 3-D models may not be available to the shape estimator).

One example of a computer vision model uses a multi-view deep neural network, where the images captured from each viewpoint would pass through its own polarization fusion backbone (see, e.g., International Patent Application No. PCT/US20/48604 filed Aug. 28, 2020, U.S. patent application Ser. No. 17/266,046, and Kalra, Agastya, et al. “Deep polarization cues for transparent object segmentation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, the entire disclosures of which are incorporated by reference herein), with independent weights for each viewpoint-specific backbone. The features computed from the images captured from each viewpoint by the separate polarization fusion backbones are then used by the computer vision model to compute a set of multi-scale features (e.g., using ResNet, as described in He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. or using a Feature Pyramid Network see, e.g., Lin, Tsung-Yi, et al. “Feature pyramid networks for object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.). The computer vision model may then compute correspondences between multi-scale features across the multiple viewpoints using a multi-view correlation search based on epipolar geometry (see, e.g., GCNet as described in Cao, Yue, et al. “Gcnet: Non-local networks meet squeeze-excitation networks and beyond.” Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019., AANet as described in Xu, Haofei, and Juyong Zhang. “Aanet: Adaptive aggregation network for efficient stereo matching.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, GA-Net as described in Zhang, Feihu, et al. “GA-Net: Guided aggregation net for end-to-end stereo matching.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019., and the like). These correspondences provide 3-D information about the scene imaged by the multi-view imaging system including cameras at different viewpoints. This 3-D information can then be used to compute surface normals and 6-DoF poses of objects in the scene. Finally, in some embodiments, the surface normals are further refined using Polarized 3D in real-time, as described in Kadambi, Achuta, et al. “Polarized 3d: High-quality depth sensing with polarization cues.” Proceedings of the IEEE International Conference on Computer Vision. 2015.

FIG. 10 is a schematic block diagram depicting a computer vision model according to some embodiments of the present disclosure. As shown in FIG. 10, in some embodiments, the computer vision model 1040 includes a first neural network 1041 (e.g., using an architecture as described above) that is trained to compute shape estimates 1060 (e.g., surface normals maps) from irradiance images 1050 captured from a scene. The computer vision model 1040 in these embodiments further include a second neural network 1042 (e.g., a convolutional neural network) or a shape-to-irradiances model 1042 that is trained to map a computed surface normals map (e.g., trained using the ground truth labels in the dataset) to synthesized image irradiances 1070 from polarization filtered cameras (e.g., estimates of what the captured polarization raw frames would give rise to the surface normals map). This second neural network 1042 may be trained by reversing the roles of the ground truth labels and the input data pairs of the training dataset. This second neural network 1042 between the actual input 1050 and the inversely predicted input 1070 may then be used by a comparison module 1080 to compute a confidence measure 1080 in the surface normals maps 1060 computed by the first neural network 1041, described above, by comparing the synthesized polarization raw frames 1070 with the observed polarization raw frames 1050 captured by the cameras. During training, the confidence measure 1090 may be used as a component of the training loss function for training the parameters of the computer vision model 1040 to increase the confidence score, and may also be used in deployment as an estimate of the confidence in the shape estimates computed by the first computer vision model 1041.

In some embodiments, the computer vision model computes estimates of the shapes of surfaces based on the polarized inverse rendering problem. In particular, the computer vision model is trained to take the AOLP, DOLP, and Intensity image from each viewpoint as input (e.g., computed from polarization raw frames captured by the cameras) and to decompose each viewpoint into polarized lighting parameters, polarized material parameters (albedo and reflectance properties), and surface normals maps. In more detail, when generating the dataset, a differentiable rendering engine may be used to in operation 850 compute the polarized lighting parameters, polarized material parameters, and surface normals maps as a part of the shape estimates, such that the rendered images rendered by the differentiable rendering engine match the appearance of the input images. In some embodiments, the model further maps the features into a material invariant polarization embedding space, thereby enabling a physics-based equation search that improves physics-based polarization reconstruction and also the accuracy of the computed surface normals maps.

Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for generating datasets for training computer vision models, such as neural networks, to predict the shapes of objects, such as the surface normals of those objects, based on a set of input images captured by an imaging system, such as a multi-view and/or multi-modal imaging system. Aspects of embodiments of the present disclosure also relate to such computer vision models trained based on such datasets. Such computer vision models provide a very efficient means of determining poses and shapes of objects, which is of value in the case of automation and robotics, where short cycle times (fast computations) increase the throughput of such systems (e.g., short processing times for determining the poses of objects enables robotic arm systems to pick and manipulate those objects more quickly, thereby enabling more objects to be manipulated per unit of time).

As noted above, in some circumstances, systems and methods for generating such datasets may be deployed within an existing production context, such as within a factory or other manufacturing facility, within a logistics pipeline (e.g., warehouse), and/or other operating environment where the shape estimator is configured to detect the shapes and/or poses of known objects. Through the process of detecting the shapes and poses of the known objects using the known 3-D model, embodiments of the present disclosure collect input images and 6-DoF poses of objects, which are then used to populate the training dataset by automatically generating ground truth data or labels for the data based on the input images, the 6-DoF poses of objects, and the 3-D models of the objects, without requiring hand labeling of these ground truth data (e.g., without requiring direct human involvement in generating these ground truth labels).

This process produces a large dataset with, possibly, millions of different images collected through the deployment of such systems in environments such as factories for autonomous manufacturing of products. Accordingly, the collected dataset can be used to train and/or re-train computer vision systems to produce robust predictions of the shapes of objects under a wide range of objects made of different materials, having different geometries (including unknown geometries), and under different illumination conditions, and therefore these systems can be quickly redeployed to new environments with little to no adaptation (e.g., retraining) required to achieve good performance, noting that additional data collected from the new environments may further improve performance.

FIG. 11 is a block diagram of a shape estimator according to one embodiment of the present disclosure. FIG. 12 is a flowchart of a method 1200 for re-training computer vision model according to one embodiment of the present disclosure. As described above with respect to FIG. 1A, in some embodiments, in operation 1210, a shape estimator 100 receives input images 18 based on images captured by the imaging system, which may include a main camera 10 and, in the case of a multi-view imaging system, support cameras 30. These input images 18 may be multi-view images (from multiple viewpoints) and may be captured using multiple imaging modalities (with or without polarization filters, in different portions of the electromagnetic spectrum, depth maps generated from time of flight or stereo, polarization signatures generated from polarization raw frames captured by camera modules with polarization filters, and the like). These input images 18 are processed by a pose estimator 120 to compute a pose of a known object in the scene in operation 1230, where a 3-D model of the known object is available to the pose estimator 120. (The pose estimator 120 may also concurrently compute the poses of multiple known objects of the same type represented by the same 3-D model or of different types represented by different 3-D models in the scene.) In the process of computing the pose of the object in the scene, the pose estimator 120 may compute a shape estimate of the object detected in the images 18 and compute a pose of the object by aligning the 3-D model to match estimated shape of the object. In some embodiments, the pose estimator 120 uses of one or more trained computer vision models to compute features from the input images 18. For example, the pose estimator 120 may use a computer vision model trained on a dataset as described above to compute a shape estimate (e.g., an estimated surface normals map of the object). The pose estimator 120 may then compare the shape estimate against a rendered shape of the 3-D model (e.g., comparing an estimated surface normals map against a rendered surface normals map) in order to update the estimated pose to reduce a difference between the rendered shape and the shape estimate as a constraint or as an additional constraint when computing the estimated pose of the object (e.g., in addition to constraints from other factors such as keypoint matching across one or more views, dense correspondence matching between rendered images and the observed surfaces of the object, and the like).

The computed pose and 3-D model of the object may then be output to a controller 28 for controlling an actuator, such as a robotic arm 24, to pick up objects detected in the input images 18. The pose and the 3-D model may also be supplied to a renderer 150 that is configured to render a final shape estimate of the object based on the 3-D model, and this shape estimate may also be supplied to the controller 28.

The computer vision model used by the pose estimator 120 to compute the shape estimate during pose estimation may also be retrained based on additional data collected from the environment in which the shape estimator 100 is operating. For example, the shape estimates generated by the renderer 150 may be combined with the observed images of the scene in operation 1250 to generate training data points where the data points include a set of one or more images and the corresponding rendered shape estimates (e.g., surface normals maps). These generated data points may then be supplied to a model trainer 170 to generate one or more data points for a dataset. The model trainer 170 may then periodically or continuously retrain the computer vision model in operation 1270 based on the additional training data (along with verifying that the updated model does not exhibit regressions or decreases in accuracy of the shape estimates). The retrained, updated computer vision model can then be installed or run by the pose estimator 120 for use in performing shape estimations as part of computing the poses of objects in received input images 18. In addition, the model trainer 170 may also receive training data (including data points of the same type as the input images 18 and ground truth shape estimates as labels) from other sources (e.g., other shape estimators deployed in other areas of the same facility or deployed in other facilities or from an external source of training data) to further update the computer vision models to improve performance. In some circumstances, the model trainer 170 is remote from the imaging system (e.g., remote from the main camera 10), such as a case where a centralized system receives training data points generated by one or more shape estimators 100, aggregates the received training data, and trains one or more computer vision models for deployment a shape estimator 100 (e.g., one or more of the shape estimators from which it received the training data).

Accordingly, shape estimators 100 in accordance with some embodiments of the present disclosure update internal computer vision models based on additional training data collected from their operational environments, thereby enabling the shape estimators 100 to continuously or periodically improve performance on the estimations of the poses and shapes of objects detected in the environment. This continuous improvement and domain adaptation is available even as the environment changes, either gradually (e.g., due to gradual changes in the types of objects presented to the system) or suddenly (e.g., deployment into a new environment with different types of objects and lighting conditions).

As such, datasets collected in accordance with aspects of embodiments of the present disclosure are useful in the training of computer vision models for performing shape estimation. The ImageNet dataset has over 14 million images that are hand-labeled to indicate what objects are pictured those images along with providing bounding box labels for those object in about one million of those images. The ImageNet dataset has had an enormous impact in improving object classification techniques over the years. Image datasets with multi-modal data (including, for example, polarization data) and corresponding ground truth labels indicating the shapes of the objects (e.g., the surface normals of surfaces depicted in the images) such as those described herein likewise enable efficient estimation of poses for new objects with new geometries and materials based on corresponding images such as their polarization and spectral signatures.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, and equivalents thereof.

Claims

1. A system for collecting data for a training dataset for training a machine learning model, the system comprising: an imaging system configured to capture one or more images; anda processing system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: for each of multiple iterations: receiving one or more respective input images of a respective scene from the imaging system;estimating a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;rendering, using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;generating a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; andincluding the respective data point in the training dataset;training the machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;obtaining one or more images of a scene;generating a shape estimate of an object in the scene from the one or more images using the machine learning model;updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; andproviding the estimated pose to a controller for causing a robot to manipulate the object.

2. The system of claim 1, wherein the imaging system comprises a polarization camera system, and wherein the one or more respective input images comprise one or more polarization images,wherein, for one or more of the multiple iterations, the operations further comprise generating a surface normals map from the one or more polarization images, andwherein aligning the keypoints in the one or more respective input images with respective corresponding keypoints of the respective 3-D model of the respective object comprises refining the respective estimated pose of the object according to differences between the surface normals map computed from the one or more polarization images and surface normals of the respective 3-D model of the object.

3. The system of claim 2 wherein the one or more polarization images comprise a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.

4. The system of claim 1, wherein the respective data point comprises one or more polarization images.

5. The system of claim 1, wherein the respective data point comprises one or more polarization signatures computed based on one or more polarization images.

6. The system of claim 1, wherein the respective data point comprises one or more surface normals maps computed from one or more polarization images.

7. The system of claim 2, wherein the respective shape estimate comprises a rendered depth map.

8. The system of claim 1, wherein the imaging system comprises a depth camera system, and wherein the respective data point comprises one or more depth maps.

9. The system of claim 8, wherein the respective pose of the respective object is estimated based on aligning a shape of the respective 3-D model with the one or more depth maps.

10. The system of claim 1, wherein for one or more of the multiple iterations, estimating the respective pose of the respective object further comprises using a previously trained machine learning model trained to compute shape estimates based on the one or more respective input images, and wherein training the machine learning model on the training dataset comprises re-training the machine learning model on the respective generated data points.

11. The system of claim 1, wherein training the machine learning model on the training dataset comprises training the machine learning model on the respective generated data points.

12. A method for collecting data for a training dataset for training a machine learning model, the method comprising: for each of multiple iterations: receiving, by a processing system comprising one or more processors and memory, one or more respective input images of a respective scene;estimating, by the processing system, a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;rendering, by the processing system using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;generating, by the processing system, a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; andincluding the respective data point in the training dataset;training the machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;obtaining one or more images of a scene;generating a shape estimate of an object in the scene from the one or more images using the machine learning model;updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; andproviding the estimated pose to a controller for causing a robot to manipulate the object.

13. The method of claim 12, wherein the one or more respective input images comprise one or more polarization images, and wherein, for one or more of the multiple iterations, the method further comprises generating a surface normals map from the one or more polarization images, wherein aligning the keypoints in the one or more respective input images with respective corresponding keypoints of the respective 3-D model of the respective object comprises refining the respective estimated pose of the respective object according to differences between the surface normals map computed from the one or more polarization images and surface normals of the respective 3-D model of the respective object.

14. The method of claim 13, wherein the one or more polarization images comprise a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.

15. The method of claim 12, wherein the respective data point comprises one or more polarization images.

16. The method of claim 12, wherein the respective data point comprises one or more polarization signatures computed based on one or more polarization images.

17. The method of claim 12, wherein the respective data point comprises one or more surface normals maps computed from one or more polarization images.

18. The method of claim 13, wherein the respective shape estimate comprises a rendered depth map.

19. The method of claim 12, wherein the respective data point comprises one or more depth maps.

20. The method of claim 19, wherein the respective pose of the respective object is estimated based on aligning a shape of the respective 3-D model with the one or more depth maps.

21. The method of claim 12, wherein, for one or more of the multiple iterations, estimating the respective pose of the respective object further comprises using a previously trained machine learning model trained to compute shape estimates based on the one or more respective input images, and wherein training the machine learning model on the training dataset comprises re-training the machine learning model on the respective generated data points.

22. The method of claim 12, wherein training the machine learning model on the training dataset comprises training the machine learning model on the respective data points.

23. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations for collecting data for a training dataset, comprising: for each of multiple iterations: receiving one or more respective input images of a respective scene;estimating a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;rendering, using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;generating a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; andincluding the respective data point in the training dataset;training a machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;obtaining one or more images of a scene;generating a shape estimate of an object in the scene from the one or more images using the machine learning model;updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; andproviding the estimated pose to a controller for causing a robot to manipulate the object.