Patent Grant
10755426
This disclosure relates generally to electronic devices comprising, or configured to use, depth sensors, cameras and the like to generate and process images and/or videos.
Digital still and/or video cameras are often incorporated in a number of electronic devices, including mobile phones, wearable devices, tablet computers, laptop computers, desktop computers, security management devices, control components of vehicles and the like. For some types of image processing applications, obtaining accurate scene depth information pertaining to different regions of an image or video frame may be important—e.g., the depth information may be used to identify objects or entities for security/safety reasons, to generate enhanced photographs in selected camera modes using a mobile phone, and so on. The depth information pertaining to a given scene may often be obtained with the help of infrared sensors and the like; for example, raw output of hardware depth sensors may be processed using low-level software or firmware to generate an initial depth map. A variety of depth sensing techniques may be used, including for example active stereo sensing, passive stereo sensing, auto-focus-based techniques, time-of-flight techniques and the like. Unfortunately, depending on aspects of the particular scene being captured, in some cases the initial depth information generated with the help of the sensors may be incomplete or inaccurate—e.g., the presence of a bright light source such as the sun may impact the accuracy of the depth information for some portions of the scene. As a result, the quality of the initial depth map may not be sufficient for at least some image processing applications.
According to some embodiments, a system may comprise a scene depth sensor, an image generator, and a depth map enhancer. The depth map enhancer may obtain an initial depth map corresponding to a scene of which a color image is produced by the image generator. The initial depth map may be based at least in part on output produced by the scene depth sensor. The depth map enhancer may determine, based at least in part on the color image and the initial depth map, initialization settings of a linear optimizer. The initialization settings may comprise respective subsets of depth indicators corresponding to a plurality of regions of the color image, including a first depth-information region which satisfies a first criterion, and a second depth-information region. The depth map enhancer may generate, using the linear optimizer, a refined depth map corresponding to the scene. The refined depth map may comprise one or more estimated depth indicators corresponding to at least the second depth-information region. Input based at least in part on the refined depth map may be provided to an image processing application.
According to one embodiment, a method may comprise obtaining, by one or more electronic devices, an image of a scene and an initial depth map corresponding to the scene. The method may include generating a refined depth map corresponding to the scene using an optimizer, the image and the initial depth map. The refined depth map may comprise one or more estimated depth indicators corresponding to at least a first depth-information region, identified based at least in part on a first criterion, in the initial depth map. Input based at least in part on the refined depth map may be provided to an image processing application.
According to some embodiments, an electronic device may comprise circuitry implementing a depth map enhancer. The depth map enhancer may obtain an initial depth map corresponding to a scene, and an image of the scene. The depth map enhancer may generate a refined depth map corresponding to the scene using an optimizer, the initial depth map and the image. The refined depth map may comprise estimated depth indicators corresponding to at least a first depth-information region of the initial depth map. The first depth-information region may be identified based at least in part on a first criterion.
This specification includes references to “one embodiment” or “an embodiment.” The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.
“Comprising.” This term is open-ended. As used in the appended claims, this term does not foreclose additional structure or steps. Consider a claim that recites: “An apparatus comprising one or more processor units . . . .” Such a claim does not foreclose the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).
“Configured To.” Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112, sixth paragraph, for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
“First,” “Second,” etc. As used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations for “first” and “second” values. The terms “first” and “second” do not necessarily imply that the first value must be written before the second value.
“Based On.” As used herein, this term is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. Any headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to. When used in the claims, the term “or” is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, as well as any combination thereof.
Embodiments of systems and methods for efficiently enhancing scene depth maps are described. According to one embodiment, a system may comprise one or more scene depth sensors, one or more image generators such as cameras, and a depth map enhancer. The scene depth sensor(s), image generator(s) and depth map enhancer may all be incorporated within a single compact device with fairly limited electrical power in some embodiments, such as a cell phone or a wearable device running at least temporarily on batteries. The depth map enhancer may obtain, with the help of the scene depth sensor(s), an initial depth map corresponding to a scene of which a color image is produced by the image generator(s). Note that at least in some embodiments, raw output produced by hardware scene depth sensors may have to be processed, e.g., using low-level firmware, software and/or hardware, to produce the initial depth map. For example, in a structured light system, during such processing, the infrared image from capture time may have its projected dot pattern matched to a reference infrared image captured earlier (at calibration) to generate the initial depth map. In some embodiments, the term “depth sensor” may be used to refer to the combination of the hardware depth sensor and the low-level processing components (if any) that are collectively responsible for generating the initial depth map. Generally speaking, in various embodiments a depth map may indicate the respective depths (approximate distances from the position where the color image was captured) of various portions of the scene (e.g., with respective depth indicators corresponding to individual pixels or groups of pixels of the image). In at least some embodiments, a given depth indicator of a depth map may comprise a single numerical value, and as such the depth indicator may also be referred to simply as a depth value; in other embodiments, more complex data structures may be used as depth indicators.
The initial depth map obtained by the depth map enhancer may comprise incomplete or inaccurate depth indicators for at least some portions of the scene in some cases, e.g., due to the presence of strong light sources, unideal reflective surfaces and the like in the scene, which may impact the quality of the output of the depth sensor. The term “hole” may be used to refer to pixels (or groups of pixels) for which the depth data included in the initial depth map is incomplete, missing or otherwise inadequate in some embodiments. An efficient linear optimizer (such as circuitry implementing a Cholesky decomposition algorithm), whose input is based at least in part on the color image and the initial depth map, may be utilized by the depth map analyzer in various embodiments to generate a refined depth map. In some embodiments, a parallel implementation of the Cholesky decomposition algorithm, optimized for execution on graphics processing units or GPUs, may be used. The refined depth map may include newly-generated depth indicators for at least some portions of the initial depth map, corresponding to one or more regions for which the initial depth map comprises inadequate depth information. Input based at least in part on the refined depth map may be provided to one or more image processing applications in some embodiments, such as image enhancement applications running on the same compact device in which the depth sensor(s), image generator(s) and the depth map enhancer are incorporated. In at least some embodiments, at least a portion of a workflow involved in generating the refined depth map may be implemented in hardware (e.g., using special-purpose chips rather than the main general purpose CPUs of the system), resulting in extremely fast refinement of the depth information at the cost of very little electrical power.
Depending on the scene, the technical advantages achieved with the help of the depth map enhancer in at least some embodiments may include a substantial improvement in depth information accuracy with respect to the initial depth image, which may in turn lead to the production of more accurate and/or more useful results by the image processing application. Thus, for example, with respect to security applications, it may become possible to identify anomalous objects/entities with a higher success rate more quickly if the depth map enhancer is used, which may in turn lead to improvements in overall safety of the environment being protected. Similarly, with respect to different photography modes supported by a cell phone at which a depth map enhancer of the kind described is used, a higher quality of photographs and/or videos may be produced as a result. For applications involving control subsystems of vehicles (such as autonomous or semi-autonomous vehicles), the use of the depth map enhancer may potentially lead to fewer accidents or other mishaps in some embodiments, as more accurate and rapid detection of objects in the vehicle's external environment may be made possible.
Note that in at least some embodiments, as described in greater detail below, the depth map enhancement algorithms described herein may rely on a number of assumptions. For example, it may be assumed in one embodiment that in a local neighborhood, regions of similar color should generally have similar depth, whereas depth discontinuities may be more likely in regions with color discontinuities. Not all of the underlying assumptions may necessarily hold true for all scenes, and the quality of the enhancement achieved may depend at least partly on the extent to which the assumptions hold. As such, in various embodiments the refined depth values maps produced by the algorithms may represent a “best estimate” of the actual depth values, with no guarantee being provided of how close the estimate is to the true or real-world depth values.
According to at least some embodiments, a pipeline or workflow comprising a plurality of phases or stages may be implemented to enhance the initial depth map. In one embodiment, for example, the input image and the initial depth map may be subdivided into regions of several different categories with respect to depth information quality, such as one or more adequate-depth-information regions, one or more background regions, and one or more inadequate-depth-information regions. As suggested by the name, with respect to the initial depth map and the pixels of the input image, an adequate-depth-information region may comprise a group of pixels for which the depth sensor(s) appear to have captured sufficient depth information for the purposes of the targeted image processing application. A background region, which may for example be designated or estimated using a mask generated for a primary subject of the image, may comprise a group of pixels for which a uniform depth indicator (e.g., the maximum permissible depth value) may be selected in some embodiments. Such a primary subject mask may be obtained, for example, using a face detection algorithm, a color propagation algorithm, or some other trained machine learning model such as a neural network-based model in different embodiments. Note that in at least some embodiments, the designation of a region as a background region may represent a best estimate or approximation; some pixels that happen to be designated as part of a background region may in fact be part of the intended primary subject or primary subject or foreground. In various embodiments, the remainder of the pixels, for which adequate depth information was not available in the initial depth map, and which are not designated as part of a background region, may be designated as inadequate-depth-information regions, for which new depth indicators are to be generated using the optimizer. In at least some embodiments, some regions of the initial depth map may already be classified as inadequate-depth-information regions by the depth sensors—e.g., special depth values representing holes may be provided for such regions in the initial depth map by the depth sensors. In various embodiments, the depth map enhancer may use one or more criteria to distinguish various types of regions from one another, such as adequate-depth-information regions versus inadequate-depth-information regions (and/or background regions). The criteria may, for example, include whether a given set of pixels was designated as a hole in the initial depth map, whether the distance of an individual pixel from an edge of an identified object exceeds a threshold, whether the distance of a bright light source from a set of pixels exceeds a threshold, and so on. The presence of a hole, or a particular distance range with respect to a bright light source or an edge, may lead to classification of a region as an inadequate-depth-information region in some embodiments, for example. The specific criteria and algorithms used for distinguishing the regions may vary in different embodiments. In some embodiments, the number of categories of regions identified may exceed the three baseline categories mentioned above (adequate-depth-information, inadequate-depth-information and background)—e.g., multiple categories of regions that do not satisfy a set of quality criteria in respective ways, and may therefore require different types of enhancements or optimization parameters, may be identified. In various embodiments, the term “depth-information regions” may be used to refer to examples of the different categories of regions identified based on the particular combination of one or more depth-related criteria being employed. Regions which meet particular criteria being used in a given embodiment may be referred to as depth-information regions of a first category or as adequate-depth-information regions, for example, while regions which do not meet the particular criteria may be referred to as depth-information regions of a second category or as inadequate-depth-information regions in such an embodiment.
In various embodiments, the workflow of depth map enhancement may include a phase in which initialization settings of a linear optimizer are generated (such as initial settings for one or more matrices of depth indicator values, constraints and weights associated with the depth indicators, and the like). Respective techniques may be used for initializing the settings for the different categories of regions in some embodiments—e.g., the original depth indicators indicated in the initial depth map may be used for adequate-depth-information regions, a constant depth indicator may be used for the background regions, while a combination of local and global depth similarity factors or formulas may be used for initializing the depth settings for inadequate-depth-information regions.
A given input image may comprise a large number of pixels (e.g., millions of pixels) in some embodiments, for individual ones of which depth information may potentially have to be generated. In at least some embodiments, in order to increase efficiency by reducing the dimensionality of the depth map refinement problem, at least some portions or stages of the enhancement workflow may be performed at the granularity of groups of pixels rather than on individual pixels. In such embodiments, from the color image, a plurality of super-pixels may be identified, where an individual super-pixel comprises one or more contiguous pixels that meet a color homogeneity criterion (and/or some other homogeneity criterion), and then depth enhancement may be performed on the super-pixels instead of on individual pixels. Super-pixels may differ from one another in size (number of constituent pixels) and/or shape in some embodiments; note that in some cases, a given super-pixel may comprise just one pixel. An aggregated (e.g., average) depth indicator for a super-pixel may be generated using a pixel-level initial depth map in various embodiments; as such, the inadequate-depth-information regions, the adequate-depth-information regions, and the background regions may each comprise depth indicators for some number of super-pixels in some embodiments. A first refined depth map generated with the help of the optimizer may comprise a respective estimated depth indicator corresponding to individual super-pixels in such embodiments. Before providing the input to the image processing application, the super-pixel-level refined depth map may be converted to a pixel-level refined depth map in various embodiments. In at least some embodiments, a filter (such as a bilateral filter or a guided image filter) may be used to obtain the pixel-level refined depth map from the super-pixel-level refined depth map.
According to one embodiment, a method may comprise obtaining an image of a scene, and an initial depth map corresponding to the scene, by one or more electronic devices. The depth may comprise incomplete depth information for at least a portion of the scene. The method may further comprise generating, using an optimizer, the image and the initial depth map, a first refined depth map corresponding to the scene, such that the first refined depth map comprises one or more estimated depth indicators corresponding to at least a first inadequate-depth-information region identified in the initial depth map. The method may also include providing input based at least in part on the first refined depth map to an image processing application. In at least some embodiments, the method may include determining a set of initialization settings of the optimizer and then invoking or executing the optimizer. The initialization settings may comprise respective subsets of depth indicators corresponding to a plurality of regions identified in the image, including one or more adequate-depth-information regions, one or more background regions, and one or more inadequate-depth-information regions including the first inadequate-depth-information region.
In some embodiments, an objective function of the optimizer used to generate or obtain the refined depth map comprise an output-to-input similarity term representing a first set of constraints, and/or a color-similarity-to-depth-similarity-correlation term representing a second set of constraints. The output-to-input similarity term may also be referred to in some embodiments as a data term, while the color-similarity-to-depth-similarity-correlation term may be referred to in some embodiments as a regularization term. In effect, the data term may help to ensure that with respect to at least some regions, the depth information generated in the refined depth map is not too different from the original depth indicators of the initial depth map in such embodiments, e.g., under the assumption that the initial depth map did contain at least somewhat useful indications of the true depths for the regions identified as inadequate-depth-information regions. The regularization term may help to ensure that, with respect to at least some regions, if two groups of pixels are similar in color (as indicated in the input image), the depth indicators corresponding to the two groups are also at least somewhat similar in the refined depth map in some embodiments. The data term may, for example, include a constraint on a difference between (a) an estimated depth indicator generated for a first group of one or more pixels by the optimizer and (b) a corresponding original depth indicator (included in the initial depth map) for the first group of pixels. The regularization term may, for example, include a constraint on a difference between (a) an estimated depth indicator generated for a first group of one or more pixels by the optimizer and (b) a depth indicator of a second group of one or more pixels, where the first group and the second group meet a color similarity criterion. In various embodiments, the operations performed to generate the refined depth map may include a set of matrix computations. According to one embodiment, the method may include determining an element of a weight matrix associated with the color-similarity-to-depth-similarity-correlation term, based at least in part on a difference in color value between two groups of pixels.
According to various embodiments, locality or proximity within the image may be used to help determine at least some initialization settings of the optimizer used for the generating the refined depth map. For example, determining the initial settings may include generating, based at least in part on a local depth similarity function or formula, a first depth indicator corresponding to a first group of one or more pixels of a first inadequate-depth-information region. The local depth similarity formula in turn may comprise a first term based on a depth of a second group of one or more pixels of an adequate-depth region, such that the second group of pixels lies within a distance threshold of the first group (e.g., at least some pixels of the second group of pixels may be contiguous to one or more pixels of the first group). Color differences may also be taken into account in the local depth similarity function in some embodiments—e.g., a second term of the local depth similarity function may be based at least in part on a difference in a color value between the first and second groups of pixels, such that the similarity in depth is reduced if the color difference is large.
Global depth similarity functions may also or instead be used in some embodiments to determine initialization depth settings for some pixels. For example, determining the initialization settings in such embodiments may comprise generating, based at least in part on a global depth similarity function, a first depth indicator corresponding to a first group of one or more pixels of the first inadequate-depth-information region. The global depth similarity function may comprise a term based on a depth of a second group of pixels of an adequate-depth-information region, where the second group is selected based on color value differences with the first group.
A number of efficiency-enhancing techniques may be used in various embodiments, e.g., to help ensure that the refined depth map is generated quickly enough for the needs of an image processing application which has stringent performance requirements. In one embodiment, in order to enable the use of a fast algorithm such as Cholesky decomposition for the optimization, one or more input matrices or vectors of the optimizer may have to meet a set of requirements. An input matrix of the optimizer may have to be positive semi-definite, and the input matrix may also have to be a square matrix. The positive semi-definite property may be attained by initializing depth indicators using the region-based approach indicated above in some embodiments. In order to ensure that the input matrix is square, in a naïve approach, matrix multiplication (involving obtaining a product of potentially very large matrices) may be required. However, in one embodiment, a shortcut technique may be employed: at least some entries of the input matrix of the optimizer may be computed directly from weight values selected for the optimizer, without performing a complete matrix multiplication. Sparsity of the input matrices may also help to increase the efficiency with which the optimization computations can be performed in various embodiments. In at least some embodiments, special-purpose chips or circuitry may be used to implement the optimizer, instead of, for example, using software running on a general-purpose CPU. In one embodiment, one or more graphics processing units (GPUs) may be used for the optimizer.
According to some embodiments, an electronic device may comprise circuitry implementing a depth map enhancer. The depth map enhancer may obtain an initial depth map corresponding to a scene and an image of the scene. The depth map enhancer may generate, using an optimizer, the initial depth map and the image, a refined depth map corresponding to the scene, where the refined depth map comprises estimated depth indicators corresponding to at least one inadequate-depth-information region of the initial depth map. Input based at least in part on the refined depth map may be provided to an image processing application. In one embodiment, a filter (such as a bilateral filter or a guided image filter) may be applied to the refined depth map to generate the input for the image processing application. The initial depth map may be obtained from a variety of depth sensors in different embodiments, such as sensors comprising infrared devices and/or LIDAR (light detection and ranging) devices.
For some scenes, in various embodiments, the initial depth map 165 may have incomplete or inaccurate depth information corresponding to at least some parts of the scene captured in the corresponding source color image (examples of factors that may cause this incompleteness/inaccuracy are discussed in further detail below with respect to
The depth map enhancer 120 may obtain the source color image 160 and the initial depth map 165 in the depicted embodiment, and analyze them with the help of an optimizer or solver 122 to help generate a refined depth map 167. The refined depth map 167 may rectify at least some of the shortcomings in the initial depth map 165 in various embodiments; e.g., various holes in the initial depth map (regions or pixels for which depth information is incomplete or inaccurate) may be filled with more accurate depth estimates. A workflow or pipeline comprising several different phases may be implemented to improve the depth information in some embodiments, including for example dimensionality reduction, region classification, initialization of settings of the optimizer, execution or invocation of the optimization algorithm, and/or dimensionality restoration. Further details regarding various aspects of the workflow are provided below, e.g., in the context of
The image processing application 130 may have fairly stringent performance requirements in at least some embodiments—e.g., the image processing may have to be completed within K milliseconds of the time that the source color image 160 was produced. The depth map enhancement workflow may accordingly be designed to be extremely efficient in various embodiments. For example, in at least some embodiments, at least a portion of the workflow, including the optimizer may be implemented using specialized hardware (e.g., one or more chips designed for optimized matrix computations or for implementing a specific optimization algorithm). In other embodiments, at least a portion of the workflow may be implemented using software run, for example, on graphical processing units (GPUs). An optimization algorithm that is known to be fast, such as a Cholesky decomposition algorithm, may be used in at least some embodiments, and various shortcut techniques may be implemented during the initialization and execution of the algorithm as discussed below in further detail.
Depending on the nature of the image processing application 130, results of the processing performed with the help of the refined depth map 167 may be stored in a repository 140 in some embodiments (e.g., a photo gallery comprising enhanced images corresponding to “portrait mode” or other modes may be populated with the results of the image processing). In some embodiments, the results may be visualized via display 150 incorporated within the electronic device 110—e.g., a user of a smart phone may be able to view an enhanced version of the source color image 160. In at least one embodiment, other types of results, which may not necessarily be limited to visible artifacts may be generated by the image processing application—e.g., with respect to security monitoring, alerts/alarms may be generated, while with respect to vehicle motion planning, vehicle trajectories may be selected and implemented.
A variety of electronic devices 110 may include a depth map enhancer 120 in different embodiments, such as smart phones, smart cameras, tablet computing devices, laptops/desktops. wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, devices incorporated within control systems of vehicles, security management/monitoring devices, Internet-of-things (TOT) devices and the like. In some embodiments the source image generator 112, the scene depth sensor 114, the depth map enhancer 130 and/or the image processing application 140 need not necessarily be implemented on, or incorporated within, the same electronic device. For example, a source image generator 112 and a scene depth sensor 114 may be part of one electronic device in some embodiments, while the depth map enhancer 130 and the image processing application 140 may be incorporated within a different electronic device.
As mentioned above, the depth information corresponding to a captured image, generated initially by one or more depth sensors, may be incomplete in some cases.
The initial depth map 240 may, for example comprise respective depth values or indicators (e.g., real numbers representing selected distance units) corresponding to at least a subset of the individual pixels (or groups of pixels) of the source image in some embodiments. Example factors 250 which may negatively influence the quality or completeness of the initial depth map 240 in various embodiments may include, for example, the presence of strong illumination sources 251, unideal reflective surfaces 252 (e.g., human or animal hair, light-absorbing objects), occlusion boundaries 253 where objects happen to be partly covered by other objects, the presence of far-distant objects 254 from which the reflected illumination is harder to detect, and so on. Each of these factors may result in initial depth estimates, for at least some portions of the scene, that are too small or too large relative to the actual or “real-world” depth of the scene portions in various embodiments; as such, the depth sensors 202 may not be able to generate depth estimates of the accuracy required or expected for an image processing application. In some embodiments, the initial depth map 240 may include best-effort values for all parts of the scene, with no specific indication of portions for which the depth data is likely to be inaccurate, and the depth map enhancer may identify at least some inadequate-depth regions of the initial depth map using one or more algorithms. In other embodiments, the depth sensor(s) 202 may themselves provide some indication of holes (pixels or groups of pixels with inadequate/unavailable depth information) in the initial depth map—e.g., negative numbers may be used as an indicator of regions for which the depth sensors were unable to judge depth accurately (in contrast to positive numbers indicating accurately-detected depth or distance). In at least one embodiment, confidence maps corresponding to the initial depth maps may be generated, indicating the estimated accuracy of the initial depth map values—for example, a value in the range 0.0 to 1.0 may be generated corresponding to each depth value in the initial depth map, with 0.0 indicating no confidence in the depth value and 1.0 indicating perfect confidence. If a confidence value exceeds a threshold, in some embodiments the corresponding pixels may be designated as part of an adequate-depth-information region. Note that although active sensing depth sensors 202 are shown by way of example in
A region-type-specific initialization phase 320 may follow the pre-processing phase in at least some embodiments. During this phase, depth values to be used for the input of the optimizer with respect to the inadequate-depth-information regions may be generated using a different algorithm than is used for the background regions and/or the adequate-depth regions, elements of one or more optimization-algorithm-dependent weight matrices and/or constraint matrices may be generated for the different regions, and so on, in the depicted embodiment. Details regarding the objective function, in summarized and matrix form, that may be employed for a particular type of optimization algorithm are provided below, e.g., in the context of
After the optimizer settings are initialized, a refined depth map may be generated by executing/invoking the optimizer in phase 325 in the depicted embodiment. In one embodiment, this phase may for example comprise Cholesky decomposition as discussed below; other types of optimization techniques may be used in other embodiments. In some embodiments, the results generated by the optimizer may be analyzed and/or transformed in a post-processing phase 330 before input 335 is provided to the targeted image processing application(s). For example, in one embodiment in which dimensionality of the inputs was reduced during pre-processing, the original dimensionality may be restored during post-processing, as indicated in element 394. In some embodiments, a different workflow or pipeline may be implemented for enhancing the initial depth map than the workflow shown in
In at least some embodiments, one or more stages or phases of the workflow 300 may be parallelized. For example, dimensionality reduction and/or restoration may be performed using multiple processors, cores or GPUs in some embodiments, with individual ones of the processing units being used for respective sub-portions of the depth map. Similarly, in at least some embodiments, the settings initialization may be performed in parallel for different parts of the depth map. In at least one embodiment, as discussed below in further detail, entries of some matrices that are part of the input of the optimizer, and may be generated naively using a potentially expensive matrix multiplication operation, may instead be generated or pre-computed without performing the complete matrix multiplication.
In the depicted embodiment, for improved efficiency, Cholesky decomposition may be performed to find the optimum depth indicator values for the output or refined depth map (thereby avoiding iterative linear solvers in such embodiments). Portions of the depth map enhancement workflow may focus on making the problem solvable using Cholesky decomposition in some embodiments, which may require A to be a square, positive semi-definite matrix. As shown in
The regularization term may be translated as constraining the depth values of pairs of output pixels to be similar to each other in the depicted embodiment—e.g., a row with a 1 at column i and a −1 at column j in the A matrix shown in matrix formulation 432 may require the output depth values of pixels i and j to be similar, based on the constraint wi,j(xi−xj)=0. The importance of the relationship between the output weights of the pair may be adjusted based on the weight value wi,j in various embodiments. In at least some embodiments, the regularization term may be used to constrain the optimizer to favor solutions with similar depths when the color values of the two pixels in the input image are similar (hence the use of the term “color-similarity-to-depth-similarity-correlation constraints”). Such similarity constraints may be important, for example, in making uniformly-colored areas at a particular distance have similar depth values, while still maintaining the flexibility to have depth discontinuities at color image object boundaries in some embodiments. It is noted that in at least some embodiments, instead of generating output depth indicators at the pixel granularity, the analysis may be performed at least initially for groups of contiguous pixels which may be referred to as super-pixels (e.g., with respect to the objective function formulations shown in
In the embodiment depicted in
The initial depth map 540 may also be generated at the pixel level in some embodiments by the depth sensor(s), and may therefore have a dimensionality of M×N. A super-pixel based depth map 560 may be generated by a depth map dimensionality reducer 550 in at least some embodiments—e.g., by averaging the initial depth values corresponding to the member pixels of the various super-pixels identified in the version 512. In at least one embodiment, if the input depth data for a sufficient number of pixels within a given super-pixel is invalid or incomplete, the depth of the super-pixel may be designated as unknown in the depth map 560. As part of the dimensionality reduction, in some embodiments, statistics about the distributions of the color values and/or depth values in individual super-pixels may be generated and stored. The optimizer 570 may operate on the super-pixel-level depth map 560, e.g., comprising K depth indicators where K<<(M×N), and generate a super-pixel level refined depth map 571 using super-pixel based constraints similar to those described above in the context of
In various embodiments, if the image processing application requires input at the pixel level, the original dimensionality may be re-imposed (e.g., for the depth map and the image) in a post-processing step 572, with a pixel level refined depth map 580 (derived from the super-pixel level refined depth map 571) being provided as input to the image processing application. One or more filters (e.g., bilateral filters and/or guided image filters) may be used during the post-processing phase in one embodiment. In some embodiments, at least some of the image processing operations may also be performed at the super-pixel granularity, so the original dimensionality may not have to be restored. It is noted that while in some of the following description, at least some stages of the depth map enhancement workflow are described as being performed at the super-pixel level, similar analysis may be performed at the individual pixel level in various embodiments; as such, super-pixel-level analysis is not a requirement for the depth map enhancement techniques described herein.
In at least some embodiments, an initial depth estimate for each super-pixel may be needed in order to obtain a positive semi-definite constraint matrix (as may be required for certain types of optimization techniques including Cholesky decomposition). The optimization problem may be formulated as a balance between a data term and a regularization term of the kind described above in the context of
The region or mask 621 may be used at one or more stages during the depth map enhancement workflow in different embodiments. For example, in at least one embodiment, one of the tasks to be performed to determine initialization settings for the optimizer may comprise estimating background regions, regions with adequate input depth information, as well as regions with inadequate depth information. In some cases, the depth sensor itself may identify regions for which high-confidence depth information has been captured, and such regions may be designated as adequate-depth-information regions. Similarly, holes or regions with clearly inadequate depth information may also be indicated in the initial depth map in some embodiments. Depending on the type of depth sensor(s) being used, estimating background regions may present a non-trivial challenge in some embodiments—e.g., for some structured infrared sensors, projected infrared dots may not be visible on distant objects, so matching to the reference infrared image may be difficult. In some such embodiments, the distance from the primary subject region (and/or an inadequate-depth-information region or hole) may be used to designate some parts of the image as background regions. In various embodiments, depth values for background region pixels or super-pixels may be set to a constant value, e.g., using a constraint of the form wxi=wbc, where bc is the constant. Elements of one or more weight matrices (such as the wi and/or wij terms used in the objective function of the optimizer) may also be selected based on whether the corresponding pixels/super-pixels are part of the primary subject region in some embodiments. Other algorithm adjustments 650 may be performed to help the optimizer achieve reasonable values for depth indicators based on assumptions/deductions about the subject matter of the image in various embodiments.
In at least some embodiments, for example because of a requirement for positive semi-definiteness of one or more of the input matrices of the optimizer, depth values may have to be selected for various holes of the initial depth map during the initialization of the optimizer. In one embodiment, local similarity and/or global similarity considerations may be taken into account when selecting such initialization-stage depth values for at least the data term of the objective or loss function. Based on local similarity considerations, in an embodiment in which super-pixel-level analysis is being performed, the initialization stage depth value di for a given super-pixel i may be estimated based on the depth values of one or more neighboring super-pixels for which adequate or valid depth information is available in the initial depth map. The initialization-stage depth value di for a super-pixel i may be set based at least in part on Local Depth Similarity Formula 1 in such embodiments.
Local Depth Similarity Formula 1 (LDSF1):
In LDSF1, ηi are the immediate neighbor super-pixels of super-pixel i, pi and pj are the respective color values of the super-pixels i and j in the input image, and σ is a tunable parameter. Small σ values may lead to sharper edges along color boundaries in various embodiments, while larger σ values may lead to smoother results and more bleeding across color boundaries. LDSF1 represents one example formula in which the depth indicators of those super-pixels that lie within a distance threshold may be taken into account, where the distance may be expressed in terms of the number of intermediary super-pixels that have to be traversed to reach super-pixel j from super-pixel i along the shortest path possible (in this approach, the distance is zero for immediate neighbors). In other embodiments, at least some non-immediate neighbors may also be considered when determining di. In embodiments in which LDSF1 is used, if the difference in color values between super-pixels i and j is high, the weight is low, and vice versa. This may make the depth propagation edge-preserving with respect to image color content in such embodiments.
After local similarity criteria are taken into account, in some cases some holes with uninitialized depth values may remain, e.g., if the holes are very different from their immediate neighbors, as may occur in the presence of point light sources and the like. For such situations, a global similarity approach may be used in some embodiments, in which non-hole super-pixels with the most similar color to a hole super-pixel are identified, and the depth of the hole super-pixel is set to the depth of such most-similar non-hole super-pixel, as per the following Global Depth Similarity Formula 1:
Global Depth Similarity Formula 1 (GDSF1):
In at least some embodiments, to maintain an edge-preserving regularization term, color similarity may also be used for setting weights wi,j, in a manner similar to that used in the local depth similarity formula LDSF1 above:
Regularization Weight Selection Formula 1 (RWSF1):
wi,j=e−|p
In embodiments in which RWSF1 is used, the regularization constraint is less powerful (has a lower weight) if the difference in color values is high. Note, however, that such color similarity metrics, while performing well generally, may not work as well within the primary subject matter region of the image some embodiments. Accordingly, in at least some embodiments, the weight may be biased higher (e.g., regardless of color differences) among neighboring super-pixels within the estimated primary subject mask region.
Using the initial depth map and the color image, the depth map enhancer may determine initialization settings (e.g., initial depth indicators for various pixels or groups of pixels, constraint matrix/vector entries, weight matrix/vector entries and the like) for a linear optimizer to be used to improve the depth map in at least some embodiments (element 707). Respective subsets of initialization-stage depth indicators may be determined for various types of depth-information regions identified/estimated in the source image, such as adequate-depth-information regions, background-regions, and inadequate-depth-information regions in various embodiments. For example, for those regions for which the initial depth map is deemed to provide correct or valid depth data, the initialization-stage depth settings may simply be copied from the initial depth map, while for regions identified/estimated as background, a constant depth indicator may be selected as the initialization-stage value. For the remaining inadequate-depth regions or holes, considerations such as local and global depth similarity of the kinds discussed earlier may be used to determine the initialization-stage depth indicators in various embodiments. In some embodiments, the optimizer may comprise specialized circuitry to generate at least some of the elements of the refined depth map (e.g., one or more special-purpose chips implementing Cholesky decomposition or a similar optimization algorithm may be used), and general-purpose CPUs of the electronic device at which the depth map enhancer is implemented may not be utilized for the optimization. In other embodiments, CPUs and/or GPUs may be used for at least some parts of the depth map enhancement workflow.
A refined depth map may be generated as output by the optimizer (element 710) in the depicted embodiment. The refined depth map may include newly-estimated depth indicators for at least regions that were identified as inadequate-depth-information regions on the initial depth map in various embodiments. If needed, post-processing operations may be performed on the refined depth map, e.g., to obtain smoothed pixel-level depth values from a super-pixel-level refined depth map in at least some embodiments (element 713). The final result of the refinement (e.g., the refined depth map, or a post-processed version thereof) may be provided as input to an image processing application in the depicted embodiment (element 716). The image processing application may utilize the improved depth data provided for various tasks in different embodiments, e.g., to generate/display enhanced versions of the input color image, to perform security-related analysis, to help plan a trajectory of a vehicle, etc.
Based on the identified super-pixels, a super-pixel-level version of the initial depth map may be generated in the depicted embodiment (element 807), reducing the dimensionality of the map from M×N to K. For example, an average of the individual depth values/indicators corresponding to the pixels of a given super-pixel may be computed to obtain the depth indicator for the super-pixel in one embodiment; other aggregation/summarization techniques may be used to determine the super-pixel-level depth data in other embodiments.
One or more operations of the depth enhancement pipeline may be implemented at the super-pixel granularity in some embodiments (element 810). For example, the identification/estimation of regions as background vs. adequate-depth-information vs. inadequate-depth-information, the identification/estimation of the primary-subject region, the initialization settings of the optimizer, and/or the execution of the optimizer may be performed at super-pixel granularity in some embodiments. In various embodiments, from a super-pixel-level refined depth map identified using the optimizer, a pixel-level refined depth map may be constructed (e.g., as part of a post-processing stage of the pipeline or workflow) and provided as input to an image processing application (element 813).
A mask indicating the estimated primary subject region may be generated in the depicted embodiment (element 904). The mask may then be used in one or more other stages of the depth map enhancement pipeline (element 907). For example, in some embodiments the mask may be used to help designate regions as background regions and/or as inadequate-depth-information regions. In one embodiment, one or more groups of pixels that are apparently located far from the primary subject mask (based on the initial depth map) may be designated as background regions and assigned a constant depth setting during initialization of the optimizer. Weight values for one or more constraints used during optimization may be set on the basis of the mask in some embodiments—e.g., so as to reduce the potential impact of color differences on depth within the primary subject region. A refined depth map generated with the help of the mask may be provided to an image processing application (element 910) in various embodiments.
In order to utilize Cholesky decomposition for determining the refined depth values, the A matrix may be required to be a square matrix as well as a positive semi-definite matrix. By selecting the initial values of x, e.g., in a region-type-dependent way as discussed earlier for adequate-depth-information regions, background regions and inadequate-depth-information regions, the positive semi-definite requirement may be met. However, because of the regularization term, the A matrix may not be square in various embodiments, and transpose operations may be used to modify the linear formulation of the problem into [WA]T[WA]x=WB, thus meeting the square matrix requirement (element 1004). If the product [WA]T[WA] is referred to as A′, and WB is referred to as B′, the linear problem to be solved may be reformulated as A′x=B′.
Obtaining the product A′ may, unfortunately, require substantial computational resources and/or power. As such, if the matrix multiplication were to be completed, more time than desired may be consumed from the perspective of the image processing application in at least some embodiments. Accordingly, a short-cut technique employing the following formulas may be used in some embodiments to compute entries of the product matrix A′ and the matrix B′ without completing a full matrix multiplication (element 1007):
where r and c are rows and columns respectively, wi is the data term weight and wi,j is the regularization weight. In addition the entries of B′ may be obtained as follows:
B′(i,1)=wi2di Shortcut Formula SF2:
where wi is the data term weight and di is the initialization-stage depth value of the data term. Because of the sparsity of the entries and the easy computation of the entries without a full matrix multiplication, the approach using the shortcut formulas SF1 and SF2 may be extremely efficient and suitable for implementation directly on hardware. In at least some embodiments, accordingly, the solution to the reformulated linear optimization problem A′x=B′ may be obtained using Cholesky decomposition implemented in hardware (element 1010).
It is noted that in various embodiments, some of the operations shown in
As mentioned earlier, a wide variety of electronic devices, such as smart phones, intelligent security cameras, automobile control system components and the like may comprise or make use of depth map enhancers in different embodiments.
In the illustrated embodiment, device 3000 includes one or more processors 3010 coupled to a system memory 3020 via an input/output (I/O) interface 3030. Device 3000 further includes a network interface 3040 coupled to I/O interface 3030, and one or more optical devices 3041, sensors 3042, and additional I/O devices 3043. Optical devices 3041 may include, for example, any combination of one or more digital or analog cameras, flashes, photodiode based ambient light sensors, scanners, and/or other optical devices. Sensors 3042 may include, for example, infrared or LIDAR based depth sensors, proximity sensors, motion detectors, orientation sensors, or other types of sensors for detecting various aspects of the electronic device environment. Depth map enhancer 3047 may, for example, include specialized circuitry that implements an efficient optimization algorithm such as Cholesky decomposition in some embodiments. I/O devices 3043 may include, for example, touch pads, touch screens, displays, keyboards, and the like. In some cases, it is contemplated that embodiments may be implemented using a single instance of electronic device 3000, while in other embodiments multiple such devices may be configured to host different portions or instances of embodiments. For example, in one embodiment some of the functionality may be implemented using one instance of an electronic device, while other portions of the functionality may be implemented using other instances of the electronic devices that are distinct from the first instance.
In various embodiments, electronic device 3000 may be a uniprocessor system including one processor 3010, or a multiprocessor system including several processors 3010 (e.g., two, four, eight, or another suitable number). Processors 3010 may be any suitable processor capable of executing instructions. For example, in various embodiments processors 3010 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors 3010 may commonly, but not necessarily, implement the same ISA. In some embodiments, graphics processing units (GPUs) may be used, e.g., for parallelizing aspects of the depth map enhancement workflow stages.
System memory 3020 may be configured to store program instructions or code 3025 and/or data 3026 accessible by processor(s) 3010. In various embodiments, system memory 3020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions 3025 may be configured to implement one or more analysis and detection modules incorporating any of the functionality described above. Additionally, data 3026 of memory 3020 may include various types of information such as user-specified settings for photography modes, and the like. In some embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory 3020 or electronic device 3000. While electronic device 3000 is described as implementing the functionality of functional blocks of previous figures, any of the functionality described herein may be implemented via such a computer system. In some embodiments, at least a portion of the workflow of depth map enhancement may be implemented by executing instructions 3025 on one or more processors 3010.
In one embodiment, I/O interface 1030 may be configured to coordinate I/O traffic between processor 3010, system memory 3020, and any peripheral devices in the device, including network interface 3040 or other interfaces, such as input/output devices 3041, 3042 or 3043. In some embodiments, I/O interface 3030 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 3020) into a format suitable for use by another component (e.g., processor(s) 3010, or depth map enhancer 3047). In some embodiments, I/O interface 3030 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface 3030 may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface 3030, such as an interface to system memory 3020, may be incorporated directly into processor 3010.
Network interface 3040 may be configured to allow data to be exchanged between device 3000 and other devices attached to a network 3050 (e.g., carrier or agent devices) or between nodes of electronic device 3000. Network 3050 may in various embodiments include one or more networks including but not limited to Local Area Networks (LANs) (e.g., an Ethernet or corporate network), Wide Area Networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments, network interface 3040 may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.
Input/output devices 3043 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, microphones, speakers, voice recognition devices, or any other devices suitable for entering or accessing data by one or more electronic devices 3000. Input/output devices 3041, 3042 or 3043 may be present in electronic device 3000 or may be distributed on various nodes of a multi-node electronic device 3000. In some embodiments, similar input/output devices may be separate from device 3000 and may interact with one or more nodes of device 3000 through a wired or wireless connection, such as over network interface 3040.
As shown in
Those skilled in the art will appreciate that electronic device 3000 is merely illustrative and is not intended to limit the scope of embodiments. In particular, the devices may include any combination of hardware or software that can perform the indicated functions, including computers, network devices, Internet appliances, PDAs, wireless phones, tablets, smart TVs, pagers, etc. Electronic device 3000 may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system. In addition, the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.
Those skilled in the art will also appreciate that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components may execute in memory on another device and communicate with the illustrated electronic device via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or a portable article to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from electronic device 3000 may be transmitted to electronic device 3000 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link. Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Generally speaking, a computer-accessible medium may include a computer-readable storage medium or memory medium such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessible medium may include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link.
The methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments. In addition, the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of claims that follow. Finally, structures and functionality presented as discrete components in the exemplary configurations may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of embodiments as defined in the claims that follow.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8443031 | Rao | May 2013 | B1 |
| 9582888 | Ramachandra et al. | Feb 2017 | B2 |
| 9633441 | Pang et al. | Apr 2017 | B2 |
| 20100046837 | Boughorbel | Feb 2010 | A1 |
| 20100208994 | Yao et al. | Aug 2010 | A1 |
| 20120120192 | Alregib et al. | May 2012 | A1 |
| 20120195492 | Ali | Aug 2012 | A1 |
| 20140146139 | Schwartz | May 2014 | A1 |
| 20140198977 | Narasimha et al. | Jul 2014 | A1 |
| 20140267610 | Ahmad et al. | Sep 2014 | A1 |
| 20140313188 | Cohen et al. | Oct 2014 | A1 |
| 20160191905 | Robinson et al. | Jun 2016 | A1 |
| 20170061701 | Mittal et al. | Mar 2017 | A1 |
| 20170124717 | Baruch | May 2017 | A1 |
| 20180061068 | Jiao | Mar 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| WO-2016176840 | Nov 2016 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20190362511 A1 | Nov 2019 | US |
