Patent Application
20240420031
The field of the disclosure relates generally to image signal processing and image perception and, more specifically, methods of enhancing machine perception.
In an image formation process, image sensor measurements are subject to degradations. Raw sensor readings suffer from photon shot noise, optical aberration, read-out noise, spatial subsampling in the color filter array (CFA), spectral cross-talk on the CFA, motion blur, and other imperfections. An image signal processor (ISP), which may be a hardware entity, addresses such degradations by processing the raw measurement in a sequential pipeline of steps, each targeting a degradation type in isolation, before displaying or saving the resulting output image. The ISP performs an extensive set of operations, such as demosaicing, denoising, and deblurring. Current image processing algorithms are designed to minimize an explicit or implicit image reconstruction loss relevant to human perceptions of image quality.
Progress in imaging and graphics has enabled many applications, including autonomous driving, automated design tools, robotics, and surveillance, where images are consumed directly by a higher-level analysis module without ever being viewed by humans. This gives rise to the question of whether signal processing is necessary, i.e., whether a learning machine is better trained directly on raw sensor data. ISPs map data from diverse camera systems into relatively clean images. However, recovering a latent image is difficult in low-light captures that are heavily degraded by photon shot noise. Low light is, in effect, a failure mode for conventional computer vision systems, which combine existing ISPs with existing classification networks.
The performance of conventional imaging and perception networks degrades under noise, optical aberrations, and other imperfections present in raw sensor data. An image-processing pipeline may interpose an image source and an image renderer to reconstruct an image that has been deteriorated. An image pipeline may be implemented using a general-purpose computer, a Field-Programmable Gate Array (FPGA), or an Application-Specific Integrated Circuit (ASIC). Conventional image-processing pipelines (ISPs) are optimized for human viewing, not for machine vision.
A demosaicing process, which is also called color-filter-array interpolation (CFA interpolation), reconstructs a full color image from incomplete color samples output from an image sensor overlaid with a CFA.
An image denoising process estimates the original image by suppressing noise from a noise-contaminated image. Several algorithms for image denoising are known in the art.
An image deblurring process attempts to remove blurring artifacts from images, such as blur caused by defocus aberration or motion blur.
It is observed that conventional perception networks, which use state-of-the-art ISPs and classifiers trained on a standard JPEG dataset, perform poorly in low light.
There is a need, therefore, to explore improved perception networks that perform well under adverse illumination conditions.
This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure described or claimed below. This description is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light and not as admissions of prior art.
The disclosure provides a novel apparatus, a learning-machine, configured for joint determination of optimal parameters of image denoising, demosaicing, and analysis. Configuration of the apparatus is based on formulating an end-to-end differentiable objective function. The apparatus accepts raw color filter array data and is flexible to handle different sensor configurations and capture settings without retraining or capturing of new training datasets.
Jointly tuning an image-reconstruction module and an image classification module outperforms training a classification module directly on raw images or the refined images produced using software and hardware Image Signal Processors (ISPs).
In accordance with an aspect, the disclosure provides a method of machine learning. The method is based on acquiring a plurality of raw images and employing at least one hardware processor to execute processes of determining a representation of a raw image of the plurality of raw images, initializing a plurality of representation parameters of the representation, defining a plurality of analysis parameters of an image analysis network configured to process the image representation, and jointly training the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function.
The process of determining a representation of a raw image starts with transforming pixel-value of the raw image to produce a variance-stabilized transformed image. The transformed image is processed in a sequence of image representation stages, each stage comprising a soft camera projection module and an image projection module, resulting in a multi-channel representation. An inverse pixel-value transformation is applied to the multi-channel representation.
The combined objective function may be formulated as a nested bilevel objective function comprising an outer objective function relevant to the image analysis network and an inner objective function relevant to the representation.
The pixel-value transformation may be based on an Anscombe transformation in which case the inverse pixel-value transformation would be an unbiased inverse Anscombe transformation. The process of pixel-value transformation also generates an added channel.
The process of image projection comprises performing steps of multi-level spatial convolution, pooling, subsampling, and interpolation. The plurality of representation parameters comprises values of the number of levels, pooling, a stride of subsampling, and a step of interpolation.
The method further comprises evaluating the learned machine using a plurality of test images and revising the number of levels, pooling parameter, a stride of the subsampling, and a step of the interpolation according to a result of the evaluation.
The method further comprises evaluating the learned machine using a plurality of test images and adding selected test images to the plurality of raw images. The processes of determining, initializing, defining, and jointly training are then repeated, thus, enabling continually updating the plurality of representation parameters and the plurality of analysis parameters.
The method further comprises cyclically operating the learned machine in alternate modes. During a first mode the plurality of raw images are updated; and the processes of determining, initializing, defining, and jointly training are executed. During a second mode, new images are analysed according to latest values of the plurality of representation parameters and the plurality of analysis parameters.
In accordance with another aspect, the disclosure provides a learning machine. The learning machine employs an image acquisition device for acquiring a plurality of raw images and comprises a memory device, and a hardware processor. The memory device comprises a plurality of storage units, storing processor executable instructions. The hardware processor comprises a plurality of processing units.
The instructions cause the hardware processor to determine a representation of a raw image of the plurality of raw images, initialize a plurality of representation parameters defining the representation, define a plurality of analysis parameters of an image analysis network configured to process the representation, and jointly train the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function.
The processor executable instructions comprise modules which cause the hardware processor to:
The processor executable instructions further comprise a module causing the hardware processor to execute an algorithm for joint optimization of nested bilevel objective functions, thereby enabling formulation of the combined objective function as an outer objective function relevant to the image analysis network and an inner objective function relevant to the representation.
The processor executable instructions further comprise a module causing the processor to implement an Anscombe transformation and a module causing the processor to implement an unbiased inverse Anscombe transformation.
The processor executable instructions further comprise a module causing the hardware processor to generate an additional channel to the transformed image.
The processor executable instructions further comprise a module causing the hardware processor to perform processes of multi-level spatial convolution, pooling, subsampling, and interpolation.
The memory device stores specified values for the number of levels, pooling parameters, a stride of subsampling, and a step of interpolation.
The processor executable instructions comprise a module causing the hardware processor to perform processes of performance evaluation using a plurality of test images; and revising the number of levels, pooling parameters, a stride of subsampling, and a step of interpolation according to a result of evaluation.
The processor executable instructions further comprise a module causing the hardware processor to perform processes of performance evaluation using a plurality of test images, adding selected test images to the plurality of raw images, and repeating the processes of determining, initializing, defining, and jointly training.
The processor executable instructions further comprise a module causing the hardware processor to perform a cyclic bimodal operation. During a first mode the plurality of raw images is updated and the processes of determining, initializing, defining, and jointly training are executed.
During a second mode, new images are classified according to latest values of the plurality of representation parameters and the plurality of analysis parameters.
Thus, the disclosure provides a learning-machine architecture for joint image reconstruction and image classification that renders classification robust, particularly under low-light conditions. A principled modular design generalizes to other combinations of image formation models and high-level computer vision tasks.
Various refinements exist of the features noted in relation to the above-mentioned aspects. Further features may also be incorporated in the above-mentioned aspects as well. These refinements and additional features may exist individually or in any combination. For instance, various features discussed below in relation to any of the illustrated examples may be incorporated into any of the above-described aspects, alone or in any combination.
The following drawings form part of the present specification and are included to further demonstrate certain aspects of the present disclosure. The disclosure may be better understood by reference to one or more of these drawings in combination with the detailed description of specific embodiments presented herein.
The following reference numerals are used throughout the drawings:
Corresponding reference characters indicate corresponding parts throughout the several views of the drawings. Although specific features of various examples may be shown in some drawings and not in others, this is for convenience only. Any feature of any drawing may be reference or claimed in combination with any feature of any other drawing.
The following detailed description and examples set forth preferred materials, components, and procedures used in accordance with the present disclosure. This description and these examples, however, are provided by way of illustration only, and nothing therein shall be deemed to be a limitation upon the overall scope of the present disclosure.
Module 120 is configured for denoising and demosaicing images in addition to performing other image improvement functions according to signal processing parameters 140. Network 130 is configured to classify an image according to the learned classification parameters 150. Upon receiving a raw image 112 from an image acquisition device 110, module 120 produces a refined image 122 which is supplied to module 130 to determine a perceived classification 132 of the raw image 112. A digital camera may save images in a raw format suitable for subsequent software processing. Thus, processes of demosaicing, denoising, deblurring may be performed to reconstruct images.
The signal processing parameters 140 and the learned classification parameters are determined independently.
Learning machine 210 comprises at least one hardware processor (not illustrated) coupled to at least one memory device storing:
The term “image analysis” refers to processes encompassing object detection, tracking, scene understanding, etc.
Upon receiving a raw image 112 from an image acquisition device 110, the image representation network 220 produces intermediate data 222 which is supplied to image analysis network 230 to determine a perceived classification 232 of the raw image 112. The intermediate data 222 comprises multiple channels.
The learned global parameters (joint parameters) 240 comprise parameters specific to the image representation network 220 and parameters specific to the image analysis network 230. Thus, learning machine 210 is configured according to joint learning of global parameters relevant to image refinement (denoising, demosaicing, . . . ) and perception (including image classification).
There are two main distinctive features of the novel learning machine 210. The first is the global optimization and the resulting global characterizing parameters. The second is the replacement of a conventional image signal processing module 120 with the image representation network 220. Referring to
Network 220 relies on repetitive activation of an image projection module 450, hereinafter referenced as module 450, which is adapted from a U-net. The U-Net is a heuristic architecture that has multiple levels, and therefore exploits self-similarity of images (in contrast to single-level architecture). A soft camera projection module 440 precedes module 450 and executes a process which permits explicit use of a color filter array (CFA) hence enabling generalization to different CFAs, or blur kernels, of different sensors. The soft camera projection module 440 together with module 450 form an image representation stage 430. The image representation stage 430 may be activated recursively (feedback loop 460). The number of turns of activation is a design choice. Alternatively, reactivation of the image representation stage may be terminated upon satisfying a specific user-defined criterion.
The raw image 112 is preferably variance stabilized prior to the repetitive activation of the image representation stage 430. Thus, the image representation network 430 employs a variance stabilizing module 420 to modify the values of pixels of the raw image 112 and a corresponding inversion module 470 to reverse the effect of initial pixel modification.
The variance stabilizing module 620 modifies the values of the pixels of a raw image 112 received from an image acquisition device 110 yielding a transformed variance stabilized image 622 and an added channel 624 as illustrated in
Thus, the image representation network 220 applies an optimization algorithm that reconstructs a latent intermediate representation from noisy, single-channel, spatially-subsampled raw measurements. In contrast to standard convolutional neural network models, the image representation network 220 renders the perception light-level independent.
The joint image representation and perception problem may be formulated as a bilevel optimization problem with an outer objective function L (classification loss function) associated with the image analysis network 230 and an inner objective function G associated with the image representation network 220. The bilevel optimization problem may be formulated as:
where Λ minimizes the inner objective function G. The output of the image representation network is a multi-channel intermediate representation Λ(y, □), which is supplied to the image analysis network 230. Here the parameters ν of the image analysis network are absorbed in L as a third argument.
Module 720 transforms a raw image 110 to a shaped image 730 so that a pixel of value p, 0≤p<pmax, is replaced with a pixel of value Ã(p); a typical value of pmax is 255. The cascade 630 (of image representation stages 430) generates multiple midway channels 750 corresponding to the shaped image 730. Module 760 offsets the effect of pixel shaping and produces a multi-channel representation 770 of a latent image to be supplied to image analysis network 230.
According to one implementation, module 720 replaces a pixel of raw image 710 of value p with a pixel of value Â(p) determined as: Â(p)=2 (p+⅜)1/2. Module 760 replaces a pixel of value q of each of the midway channels 750 with a pixel of value Ä(q) determined as:
Alternative variance stabilizing transforms Ã(p) and corresponding inverse transforms Ä(q) are known in the art.
The contracting path is a convolutional network where application of two 3×3 unpadded convolutions is repeated. A rectified linear unit (ReLU) and a 2×2 max pooling operation with stride 2 for downsampling succeed each convolution. At each downsampling, the number of feature channels is doubled.
In the expanding path, an upsampling process of the feature map is followed by a 2×2 convolution that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3×3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in every convolution. At the final layer a 1×1 convolution is used to map each multi-component feature vector to the desired number of classes.
A soft camera projection process 440 is applied to an output 1010 of the variance stabilizing module 620 or output of a preceding activation of an image projection module (activation of a U-Net stage).
Processes 1000 of image projection module 450 (a single U-Net stage) include:
According to a first spatial convolution scheme, a window 1140 of pixels of a filter slides within the m×n pixels so that the filter is completely embedded thus yielding a feature map 1150 of dimension (m−w+1)×(n−w+1) pixels. According to a second spatial convolution scheme, the window of pixels of the filter slides within the m×n pixels so that the intersection region exceeds Δ×Δ pixels, 0<Δ<w, yielding a feature map 1160 of dimension (m−Δ+1)×(n−Δ+1) pixels.
Process 1540 executes the image projection module (a U-Net stage) 450 to determine an image representation. Process 1542 determines whether further activation of processes 1530 and 1540 are beneficial. The decision of process 1542 may be based on a predefined criterion. However, in order to facilitate end-to-end optimization to jointly determine optimal parameters of module 450 and weights of the image analysis network 230, it is preferable to predefine the number of cycles of executing process 1530 and 1540 where the parameters may differ from one cycle to another. A conjectured preferred number of cycles is eight. Process 1550 performs an unbiased inverse transform to offset the effect of pixel shaping of process 1520. Process 1520 may be based on the Anscombe transform, in which case process 1550 would be based on an unbiased inverse Anscombe transform as illustrated in
The invention provides an end-to-end differentiable architecture that jointly performs demosaicing, denoising, deblurring, tone-mapping, and classification. An end-to-end differentiable model performs end-to-end image processing and perception jointly.
The architecture illustrated in
A memory device storing a training module 1720 comprising software instructions, a memory device storing training images 1730, and a memory device 1740A are coupled to processor 1710 forming a training segment 1741 of the learning system. A memory device storing an image analysis network 1760 comprising software instructions, a buffer storing incoming images 1770 to be analyzed and classified, and a memory device 1740B are coupled to processor 1750 forming an operational segment 1742 of the learning system which determines a classification (a label) for each incoming image.
The training segment 1741 produces continually updated learned global parameters (joint parameters) which are stored in memory device 1740A. The learned global parameters may be transferred, through an activated link 1743, to memory device 1740B periodically or upon completion of significant updates.
The training segment 1741 (first mode) relates to end-to-end training. The operational segment 1742 (second mode) relates to actual use of the trained machine. Alternatively, the learning machine may be operated in a cyclic time-multiplexed manner to train for a first period and perform perception tasks, for which the machine is created, during a second period. Thus, the learning machine may perform a cyclic bimodal operation so that during a first mode the training images 1730 are updated and the training module 1720 is executed, and during a second mode, new images 1770 are analyzed and classified according to latest values of learned parameters.
Thus, an improved method and system for machine learning have been provided. The method of machine learning is based on acquiring a plurality of raw images and employing at least one hardware processor to execute processes of determining a representation of a raw image of the plurality of raw images, initializing a plurality of representation parameters of the representation, defining a plurality of analysis parameters of an image analysis network configured to process the image representation, and jointly training the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function. The combined objective function may be formulated as a nested bilevel objective function comprising an outer objective function relevant to the image analysis network and an inner objective function relevant to the representation.
The process of determining a representation of a raw image starts with transforming pixel-value of the raw image to produce a variance-stabilized transformed image. The transformed image is processed in a sequence of image representation stages, each stage comprising a soft camera projection module and an image projection module, resulting in a multi-channel representation. An inverse pixel-value transformation is applied to the multi-channel representation. The pixel-value transformation may be based on an Anscombe transformation in which case the inverse pixel-value transformation would be an unbiased inverse Anscombe transformation. The process of pixel-value transformation also generates an added channel.
The process of image projection comprises performing steps of multi-level spatial convolution, pooling, subsampling, and interpolation. The plurality of representation parameters comprises values of the number of levels, pooling, a stride of subsampling, and a step of interpolation.
The learned machine may be evaluated using a plurality of test images. The number of levels, pooling parameter, a stride of the subsampling, and a step of the interpolation may be revised according to a result of the evaluation. Selected test images may be added to the plurality of raw images then the processes of determining, initializing, defining, and jointly training would be repeated.
The learned machine may be cyclically operated in alternate modes. During a first mode the plurality of raw images are updated and the processes of determining, initializing, defining, and jointly training are executed. During a second mode, new images are analysed according to latest values of the plurality of representation parameters and the plurality of analysis parameters.
Some embodiments involve the use of one or more electronic processing or computing devices. As used herein, the terms “processor” and “computer” and related terms, e.g., “processing device,” “computing device,” and “controller” are not limited to just those integrated circuits referred to in the art as a computer, but broadly refers to a processors, a processing device, a controller, a general purpose central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a microcomputer, a programmable logic controller (PLC), a reduced instruction set computer (RISC) processor, a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), and other programmable circuits or processing devices capable of executing the functions described herein, and these terms are used interchangeably herein. These processing devices are generally “configured” to execute functions by programming or being programmed, or by the provisioning of instructions for execution. The above examples are not intended to limit in any way the definition or meaning of the terms processor, processing device, and related terms.
The various aspects illustrated by logical blocks, modules, circuits, processes, algorithms, and algorithm steps described above may be implemented as electronic hardware, software, or combinations of both. Certain disclosed components, blocks, modules, circuits, and steps are described in terms of their functionality, illustrating the interchangeability of their implementation in electronic hardware or software. The implementation of such functionality varies among different applications given varying system architectures and design constraints. Although such implementations may vary from application to application, they do not constitute a departure from the scope of this disclosure.
Aspects of embodiments implemented in software may be implemented in program code, application software, application programming interfaces (APIs), firmware, middleware, microcode, hardware description languages (HDLs), or any combination thereof. A code segment or machine-executable instruction may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to, or integrated with, another code segment or a electronic hardware by passing or receiving information, data, arguments, parameters, memory contents, or memory locations. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the claimed features or this disclosure. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description herein.
When implemented in software, the disclosed functions may be embodied, or stored, as one or more instructions or code on or in memory. In the embodiments described herein, memory includes non-transitory computer-readable media, which may include, but is not limited to, media such as flash memory, a random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and non-volatile RAM (NVRAM). As used herein, the term “non-transitory computer-readable media” is intended to be representative of any tangible, computer-readable media, including, without limitation, non-transitory computer storage devices, including, without limitation, volatile and non-volatile media, and removable and non-removable media such as a firmware, physical and virtual storage, CD-ROM, DVD, and any other digital source such as a network, a server, cloud system, or the Internet, as well as yet to be developed digital means, with the sole exception being a transitory propagating signal. The methods described herein may be embodied as executable instructions, e.g., “software” and “firmware,” in a non-transitory computer-readable medium. As used herein, the terms “software” and “firmware” are interchangeable and include any computer program stored in memory for execution by personal computers, workstations, clients, and servers. Such instructions, when executed by a processor, configure the processor to perform at least a portion of the disclosed methods.
Several terms used in the detailed description are commonly used in the art. See, for example, references shown below, all of which are incorporated herein by reference.
As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural elements or steps unless such exclusion is explicitly recited. Furthermore, references to “one embodiment” of the disclosure or an “exemplary embodiment” are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Likewise, limitations associated with “one embodiment” or “an embodiment” should not be interpreted as limiting to all embodiments unless explicitly recited.
Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is generally intended, within the context presented, to disclose that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Likewise, conjunctive language such as the phrase “at least one of X, Y, and Z,” unless specifically stated otherwise, is generally intended, within the context presented, to disclose at least one of X, at least one of Y, and at least one of Z.
The disclosed systems and methods are not limited to the specific embodiments described herein. Rather, components of the systems or steps of the methods may be utilized independently and separately from other described components or steps.
This written description uses examples to disclose various embodiments, which include the best mode, to enable any person skilled in the art to practice those embodiments, including making and using any devices or systems and performing any incorporated methods. The patentable scope is defined by the claims and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences form the literal language of the claims.
This application is a Continuation of U.S. patent application Ser. No. 18/488,010 filed Oct. 16, 2023, which is a Continuation of U.S. patent application Ser. No. 17/850,785 filed Jun. 27, 2022, now U.S. Pat. No. 11,809,975, which is a Continuation of U.S. patent application Ser. No. 17/843,174 filed Jun. 17, 2022, now U.S. Pat. No. 11,790,272, which is a Continuation of U.S. patent application Ser. No. 17/712,727 filed Apr. 4, 2022, now U.S. Pat. No. 11,783,231, which is a Continuation of U.S. patent application Ser. No. 16/927,741 filed Jul. 13, 2020, now U.S. Pat. No. 11,295,176, which is a Continuation of U.S. patent application Ser. No. 16/025,776 filed Jul. 2, 2018, now U.S. Pat. No. 10,713,537, which claims the benefit of U.S. Provisional Patent Application No. 62/528,054 filed on Jul. 1, 2017, the entire contents of which are hereby incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62528054 | Jul 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18488010 | Oct 2023 | US |
| Child | 18821683 | US | |
| Parent | 17850785 | Jun 2022 | US |
| Child | 18488010 | US | |
| Parent | 17843174 | Jun 2022 | US |
| Child | 17850785 | US | |
| Parent | 17712727 | Apr 2022 | US |
| Child | 17843174 | US | |
| Parent | 16927741 | Jul 2020 | US |
| Child | 17712727 | US | |
| Parent | 16025776 | Jul 2018 | US |
| Child | 16927741 | US |
