Patent Application
20220215505
The present disclosure relates generally to image capture and processing. More particularly, the present disclosure relates to a system and method of processing of a captured image to facilitate post-processing modification.
When a digital image is captured, for example with a digital camera, several image processing operations are generally performed to convert the captured scene radiance into an appropriate image; for example, into a visually pleasing image. An image processor (IP) located in the camera, or located remote to the camera, performs these operations in a sequential image processing pipeline to convert the scene-referred image I recorded by the sensor to the final output-referred image O produced by the IP. Generally, as the input image I is passed through the image processing operations in the pipeline, each operation generally alters the image until the output image O is produced; in this way, the intermediary images and any changes applied from each operation are discarded and not retained. Thus, the output image O is dependent on the settings used by the IP.
In an aspect, there is provided a method of processing of a captured image to facilitate post-processing modification, the method executed on one or more processors, the method comprising: receiving the captured image; down-sampling the captured image to generate a down-sampled image; passing the captured image through an image processing pipeline to generate an initial output image, the image processing pipeline comprising performing one or more image processing operations on the passed image based on a set of parameters; iteratively passing, in one or more iterations, the down-sampled captured image through the image processing pipeline to generate an output down-sampled image for each iteration, each iteration using at least one variation to the set of parameters used by the image processing pipeline; and storing the initial output image and the one or more output down-sampled images.
In a particular case of the method, the method further comprising: associating the one or more output down-sampled images with the initial output image; and outputting the initial output image with the associated one or more output down-sampled images.
In another case, of the method associating the one or more output down-sampled images with the initial output image comprises storing the one or more output down-sampled images as metadata to the initial output image.
In yet another case of the method, the method further comprising storing the down-sampled captured image as metadata to the initial output image.
In yet another case of the method, the method further comprising determining a mapping between the down-sampled captured image and the captured image, and storing the mapping as metadata to the initial output image.
In yet another case of the method, one of the image processing operations comprises performing white-balancing, and wherein the variation in the set of parameters comprises variations of color temperature for the white-balance.
In yet another case of the method, the method further comprising: down-sampling the initial output image to generate a down-sampled initial output image, the down-sampled initial output image comprising dimensions equivalent to the one or more output down-sampled images; and for each of the output down-sampled images, determining a mapping between the down-sampled initial output image and the respective output down-sampled image.
In yet another case of the method, associating the one or more output down-sampled images with the initial output image comprises storing the mapping for each of the output down-sampled images as metadata to the initial output image.
In yet another case of the method, the mapping comprises a nonlinear mapping function that minimizes error between colors in the down-sampled initial output image and the respective output down-sampled image comprising the at least one variation to the set of parameters used by the image processing pipeline.
In yet another case of the method, the method further comprising using a kernel function to transform red, green and blue (RGB) triplets of the down-sampled initial output image to a dimensional space that is greater than three dimensions, the minimization of error between the colors comprising minimizing a squared-distance between the down-sampled initial output image in the dimensional space and the respective output down-sampled image.
In yet another case, the kernel function output comprising: (R, G, B, R2, G2, B2, RG, RB, GB, R3, G3, B3, RG2, RB2, GB2, GR2, BG2, BR2, RGB, R4, G4, B4, R3G, R3B, G3R, G3B, B3R, B3G, R2G2, G2B2, R2B2, R2GB, G2RB, B2RG)T.
In yet another case of the method, the method further comprising: associating the one or more mappings with the initial output image by storing the one or more output down-sampled images as metadata to the initial output image; and outputting the initial output image with the associated one or more mappings.
In yet another case of the method, the method further comprising: generating a modified output image by applying one of the one or more mappings to the initial output image; and outputting the modified output image.
In another aspect, there is provided a method of generating one or more modified output images from a captured image, the method executed on one or more processors, the method comprising: receiving an initial output image processed using an image processing pipeline from the captured image, the image processing pipeline comprising one or more image processing operations on the captured image based on a set of parameters; receiving one or more mappings associated with the initial output image, each of the one or more mappings having been determined from a respective mapping between a down-sampled initial output image and one of one or more output down-sampled images, the one or more output down-sampled images having been determined by iteratively passing, in one or more iterations, a down-sampled image of the captured image through the image processing pipeline, each iteration using at least one variation to the set of parameters used by the image processing pipeline; generating each of the one or more modified output images by applying one of the one or more mappings to the initial output image; and outputting at least one of the modified output images.
In yet another aspect, there is provided a system of processing of a captured image to facilitate post-processing modification, the system comprising one or more processors and one or more non-transitory computer storage media, the one or more non-transitory computer storage media comprising instructions that cause the one or more processors to execute: an input module to receive the captured image; a down-sampling module to down-sample the captured image to generate a down-sampled image; and an image processing module to pass the captured image through an image processing pipeline to generate an initial output image, the image processing pipeline comprising performing one or more image processing operations on the passed image based on a set of parameters, and to iteratively pass, in one or more iterations, the down-sampled captured image through the image processing pipeline to generate an output down-sampled image for each iteration, each iteration using at least one variation to the set of parameters used by the image processing pipeline.
In a particular case of the system, the one or more processors further execute an output module to associate the one or more output down-sampled images with the initial output image, and to output the initial output image with the associated one or more output down-sampled images.
In another case of the system, associating the one or more output down-sampled images with the initial output image comprises storing the one or more output down-sampled images as metadata to the initial output image.
In yet another case of the system, the down-sampling module further down-samples the initial output image to generate a down-sampled initial output image, the down-sampled initial output image comprising dimensions equivalent to the one or more output down-sampled images, and the one or more processors further execute a mapping module to, for each of the output down-sampled images, determine a mapping between the down-sampled initial output image and the respective output down-sampled image.
In yet another case of the system, the mapping comprises a nonlinear mapping function that minimizes error between colors in the down-sampled initial output image and the respective output down-sampled image comprising the at least one variation to the set of parameters used by the image processing pipeline.
In yet another case of the system, the mapping module further uses a kernel function to transform red, green and blue (RGB) triplets of the down-sampled initial output image to a dimensional space that is greater than three dimensions, the minimization of error between the colors comprising minimizing a squared-distance between the down-sampled initial output image in the dimensional space and the respective output down-sampled image.
In yet another case of the system, the one or more processors further execute an output module to associate the one or more mappings with the initial output image by storing the one or more output down-sampled images or the mappings as metadata to the initial output image, and to output the initial output image with the associated one or more mappings.
In yet another case of the system, the mapping module further generates a modified output image by applying one of the one or more mappings to the initial output image, and the one or more processors further execute an output module to output the modified output image.
These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of systems and methods to assist skilled readers in understanding the following detailed description.
The features of the invention will become more apparent in the following detailed description in which reference is made to the appended drawings wherein:
Embodiments will now be described with reference to the figures. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.
Various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: “or” as used throughout is inclusive, as though written “and/or”; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender; “exemplary” should be understood as “illustrative” or “exemplifying” and not necessarily as “preferred” over other embodiments. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.
Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.
The following relates generally to image capture and processing. More particularly, the following relates to a system and method of processing of a captured image to facilitate post-processing modification.
While the following describes a scene-referred image I (also referred to as a sensor image or captured image) captured by a camera, it is to be understood that any suitable image capture device and/or sensor can be used; for example, a video-camera, a webcam, a camera capturing light outside of the visible spectrum, or the like.
An image captured by a camera's sensor can be referred to as a scene-referred image I or a “raw” image. After an image is captured, several image processing operations can be performed to convert the scene-referred image I into an appropriately processed image; for example, such that the image is visually pleasing. An image processor (IP) can perform these image processing operations by following a given image processing pipeline. The image processing pipeline is thus used to convert the scene-referred image I recorded by the camera's sensor to the final output-referred image O (also referred to generally as the output image). Generally, as the input image I is passed through the image processing operations in the pipeline, each operation alters the image until the output image O is produced; in this way, the intermediary outputs and/or images from each operation are not retained.
The output-referred image O 304 is generally saved in an output-referred color space, for example, standard RGB (sRGB), AdobeRGB, YUV, or the like. The output image may be stored locally or uploaded to an external database or cloud storage. The output image may be compressed and stored in a format such as JPEG. In this way, output-referred image O 304 represents the scene-referred image I 302 processed by the IP pipeline 310 with settings j.
With the approach illustrated in
While some cameras provide users the ability to save the original raw image data recorded by the sensor, this ability is not always available and comes at a very high cost as far as file sizes and storage requirements. To overcome this limitation, some approaches attempt to linearize the sRGB output image. This process, called radiometric calibration, is closely related to raw image reconstruction. The objective of radiometric calibration is to undo the non-linear processing applied by the IP to produce the final sRGB output. Generally, radiometric calibration is a tedious procedure where a system needs to image many color charts under different lights to compute the necessary mathematical operators to perform the reversal of the IP processing.
These mathematical operators change with different settings on the camera, so the system would need to have several different mathematical models for each setting. Further, the system generally has to store this information somewhere and have it ready in the event it needs to use it. In addition, this information generally needs to work in both directions, one to reverse the output image O back to the scene-referred image I. And then another, to map the scene-referred image I back to the output image O after the modified setting is applied. Radiometric calibration to invert the IP processing has been around for a long time, but its uptake is extremely limited because it requires too much effort and storing the additional data for each image is not practical.
Other approaches use raw reconstruction techniques to convert sRGB values back to their original raw-RGB values. For example, raw-RGB reconstruction based on a guided up-sampling of a low-resolution raw image, stored along with the sRGB image, to reconstruct the full-resolution raw image. Further approaches use more complex models for the onboard camera processing stages to achieve higher raw reconstruction accuracy; however, this requires either a small raw file to be stored along with the sRGB image or involves careful camera calibration. Similar to radiometric calibration, such camera calibration generally requires the system to know how to go back to the scene-referred image I, compute modified setting, and then apply the IP again to get the corrected output image O. Advantageously, the present embodiments do no require such extra computation and storage of having to reverse the IP on the output image O to get back to the scene-referred image I.
Generally, the other approaches require attempting to get back to the raw-RGB image I 302 and then re-process the image again through the image processing pipeline 310, or some variant of the image processing pipeline. These approaches are thus computationally expensive and require access to the image processing pipeline, which may not be available if the output image O is received on a different computing device. Advantageously, the present embodiments do not require going back from the sRGB image O to the raw image I and forward again to the sRGB image O. Instead, the present embodiments provide the technical advantage of being able to perform processing of the rendered sRGB images in the sRGB color space. As described, the results of the present embodiments can appear effectively identical to an output image O if the sensor image I were processed by the image processing pipeline using a different setting.
Turning to
In an embodiment, the system 100 can be located on an image capture device 106; such as a camera. In this case, the system can be implemented, for example, with general or specialized computing components, or with a system-on-chip (SoC) implementation. In other cases, the system 100 can be located separate or remote from the image capture device 106. In this case, the system 100 may receive the data comprising the scene-referred image I 302 via a communication link or network, for example, the Internet.
Advantageously, the present embodiments overcome substantial challenges in the art by, for example, providing an approach that significantly facilitates post-processing correction of images incorrectly or inopportunely processed by an image processing pipeline, such as those in a camera. For example, it can be a substantial advantage that the present embodiments can be used to produce output images that appear almost identical to what the image processing pipeline would have produced if different settings j were applied.
As illustrated in
Advantageously, the resulting down-sampled images {ti}Ni=1 409 provide sufficient information to estimate mathematical operators {Mi}Ni=1 that allow the RGB values in the image O 304 to be transformed to the RGB values in {ti}Ni=1. The operators {Mi}Ni=1 in turn allow for the transformation of color values (post-processing) in image O that was captured with settings j such that it effectively appears as if image O were originally processed with the image processing pipeline using settings {ki}Ni=1.
In further cases, the mapping operators {Mi}Ni=1 can be stored in the metadata associated with the image O instead of the resulting down-sampled images {ti}Ni=1 409 as described above. In this case, the values representing {Mi}Ni=1 will generally require even less storage than the resulting down-sampled images {ti}Ni=1 409. Advantageously, the present embodiments can require minimal metadata to be embedded or otherwise associated with the image O. Thus, the present embodiments can store metadata necessary to provide post-processing on each image O such as to appear is if it was captured with different camera settings. This is a substantial technical improvement over the art because, for example, generally there were very limited and mostly ineffective post-processing that could be done to an output image 304 after it was already processed by the image processing pipeline. For example, other attempts to conduct white-balance correction post-hoc on the output image O are generally ineffective; for example, because the IP can work at a higher bit-depth prior to doing conversion to an image color-space, because the IP generally operates on linear values rather than gamma values, and because of the accumulation of errors when performing post-hoc white-balancing.
Turning to
At block 202, the input module 150 receives a sensor image I. For example, from a camera sensor on an image acquisition device 106 or from a database 146. In some cases, the sensor image I can be referred to as a “raw image.”
Blocks 204 to 208 generally illustrate the generation of an output image with associated down-sampled images. At block 204, the down-sampling module 152 generates a down-sampled image s of the sensor image I. The down-sampled image s having a fraction of the resolution of the sensor image I; for example, 1/50th the resolution of the sensor image I.
The down-sampling module 152 can use any suitable approach to down-sample images. For example, performing an interpolating or weighted averaging function that averages pixels in a given window that is related to the size of the down sampled image; in this case, effectively the average color in this window is recorded for the down-sampled image. In a particular example, nearest-neighbor interpolation can be used that down-samples the image by taking samples in either a grid structure or in a more complex structure (e.g., Voronoi diagram). Other more complex algorithms may be used for interpolation in order to reduce the artifacts in the down-sampled image (e.g., bilinear and bicubic interpolations). Other down-sampling techniques include: Box sampling, Mipmap, Fourier-transform-based methods, deep convolutional neural networks (CNNs), pixel skipping, or the like.
At block 206, the image processing module 154 performs one or more operations on the sensor image I by passing the sensor image I through the operations of an image processing pipeline. The output of the image processing pipeline is an output image O; for example, an sRGB image. The output image O is then stored in memory. Each of the operations of the image processing pipeline using one or more parameters; the set of parameters collectively referred to as j.
At block 208, the image processing module 154 iteratively, for one or more N iterations, performs the one or more operations of the image processing pipeline on the down-sampled image s. With each iteration i, the image processing module 154 changes at least one of the parameters of at least one of the operations of the image processing pipeline; each changed set of parameters referred to as ki, where {ki}Ni=1. In an example, the image processing module 154 could iteratively process the down-sampled image s with different pre-set values for the white-balance parameter: tungsten, fluorescent, daylight, and the like. In further examples, the differing parameters can include sharpening operations turned off, noise reduction operations turned off, different color rendering/encoding paradigms, or the like. The output of each iteration of the image processing pipeline is an output down-sampled image ti, where {ti}Ni=1 represents the total set of output down-sampled images. The set of output down-sampled images {ti}Ni=1 can be stored in memory separately or associated with the output image O, for example, as metadata associated with the output image O. In further cases, set of output down-sampled images {ti}Ni=1 can be stored as non-displayed pixels associated with the output image O.
In some cases, one of the output down-sampled images ti can include just the down-sampled image s without any operations of the image processing pipeline applied (equivalent to having the pipeline operations disabled), or with only a selected subset of the pipeline operations applied. In further cases, the metadata can include the mapping operator Mi from the down-sampled image s to the sensor image I. Thus, in either case, the respective mapping operator Mi can be applied to the down-sampled image s to arrive back at the sensor image I. Advantageously, this allows the raw image to be retained for post-processing, such as by photo editing software, without requiring saving the large file size raw image itself.
Blocks 210 to 214 generally illustrate acquiring a modified output image having changed properties based on different parameters. For example, changing output image O to appear as if it was processed with one of the changed parameter sets ki. In order to arrive at the modified output image Omodified, the mapping module 156 determines one or more color mapping operators Mi.
At block 210, the down-sampling module 152 down-samples the initial output image O to have the same pixel dimensions as the set of output down-sampled images {ti}Ni=1. The down-sampled initial output image O, with parameter set j, can be referred to as Osmall.
At block 212, for each output down-sampled image ti, the mapping module 156 determines a nonlinear mapping function Mi: μ→
3, which maps Osmall to ti, by determining the following minimization:
where Φ: 3→
μ is a kernel function that transforms the RGB triplet to a μ-dimensional space, and where μ>3 and ∥·˜IIF denotes the Frobenius norm. For each output down-sampled image ti, the above determination finds a mapping, Mi, that minimizes the errors between the RGB colors in the down-sampled initial output image Osmall and its corresponding output down-sampled image ti. The set of mappings Mi can be stored in memory separately or associated with the output image O, for example, as metadata associated with the output image O. In some cases, where the set of output down-sampled images {ti}Ni=1 are stored as metadata, the mappings, Mi, can be determined when the changed output image O is required to be determined. In other cases, where the mappings, Mi, are stored as metadata associated with the initial output image O, the set of output down-sampled images {ti}Ni=1 can be discarded after the mappings are determined; for example, when the output image O and its associated metadata are committed to memory. In an example, metadata can include storing the mappings, Mi, is comment fields associated with JPEG images.
It is contemplated that various suitable kernel functions and dimensions of the a μ-dimensional space may be used. As an example, TABLE 1 shows examples of different kernel functions that can be used. The first column represents the dimensions of the output vector of the corresponding kernel function in the second column. The term PK refers to a polynomial kernel, while the term RPK refers to root polynomial kernel.
In some cases, relying on kernel functions with higher degrees could hinder the generalization, often referred to as overfitting. In the present embodiments, the mapping Mi is determined specifically for each pair of images. Hence, a kernel function with a higher degree may be preferable. In some cases, once the mappings Mi are determined, the set of down-sampled output images, ti may no longer be needed, and can thus be discarded. In this case, advantageously, only the mappings, having low resource requirements, are required to be retained. In general, nonlinear operations are applied during rendering. Because of these nonlinearities, lower kernel functions (having less polynomials) or linear 3×3 mapping operators may lead to underfitting; such that the mapping operator may not be the best fitting and capture the underlying trend of the source and target data. A higher degree for the kernel functions generally provide the ability to find the best fit to map the source image to the target image; however, they often require more computational time compared with kernel function with a lower degree. In the present inventors' example experiments, the kernel function 34 (PK) was determined to have the best performance characteristics in terms of computational time and fitting accuracy. In other cases, other suitable kernel functions and other suitable mappings Mi based on Equation (1) can be used.
At block 214, the mapping module 156 can use the mappings Mi to facilitate post-processing of the initial output image O to generate the modified output images Omodified. To determine each of the modified output images Omodified, the mapping module 156 can use
O
modified
=M
iΦ(O) (2)
where Omodified is the full-resolution modified output image as if it was “re-processed” with the ith parameter ki.
At block 216, the output module 158 stores or otherwise outputs one or more of the modified output images Omodified; for example, outputting to the output interface 136, or outputting for storage in the database 146, non-volatile storage 144, RAM 128, or a remote location. In some cases, the modified output images can be determined at a later time. In these cases, the output module 158 can output the initial output image O with the associated output down-sampled images {ti}Ni=1 , or it can output the initial output image O with the associated mappings Mi.
While the present embodiments describe generating output down-sampled images, it is understood that any representation can be used so long as the pixel values of the down-sampled images are retained; for example, histogram representations can be used.
As an example experimental implementation of the present embodiments,
Qualitative example evaluations conducted by the present inventors illustrate that the present embodiments can be used to apply post-processing modifications, for example, fix incorrectly white-balanced images or to produce visually pleasing variations of the image, with a reasonably small reconstruction error. For the example evaluations, for each image, the image processing pipeline down-sampled the demosaiced raw image into 150×150 pixels in order to generate five down-sampled images with the following color temperatures: 2500, 4000, 5500, 7000, and 8500. 34 (PK) was used as the kernel function. The mapping operators were represented as 34×3 matrices. As a result, the modified output images corrected white-balance in a way that was not reasonably distinguishable to the naked eye to white-balance corrected images that had been processed by the camera's image processing pipeline.
Advantageously, the present embodiments provide the ability to use a down-sampled version of a captured raw image with different parameters to facilitate post-processing modification. The facilitation allows for an improvement in performance without requiring many modifications to the existing camera imaging pipelines. By using the down-sampled output images, the required computational time to compute the mapping function is very small and insignificant. Advantageously, the present embodiments can be incorporated into current camera imaging pipelines with relatively minimal modifications. Additionally, a stand-alone implementation can be used where RAW image capture is available.
Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto. The entire disclosures of all references recited above are incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CA2020/050465 | 4/9/2020 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 62831442 | Apr 2019 | US |
