Patent Application
20240020808
This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2022-0087068, filed on Jul. 14, 2022, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.
The following description relates to a method and apparatus with image restoration.
Image restoration may involve restoring an image of degraded quality to an image of improved quality. Image restoration may be performed by a deep learning-based neural network. The neural network may be trained based on deep learning and may perform inference for a desired purpose by mapping input data and output data that are in a nonlinear relationship to each other. Such a trained capability of generating the mapping may be referred to as a learning ability of the neural network. The neural network trained for a special purpose, such as image restoration, may have a general ability to generate a relatively accurate output in response to an input pattern that is not yet trained.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In one general aspect, an image restoration method includes determining auxiliary data corresponding to a plurality of filter kernels by filtering target data with the plurality of filter kernels, determining new input data by combining the auxiliary data with at least some input data of layers of a neural network-based image restoration model, and generating, based on the new input data, a restored image of an input image by executing the neural network-based image restoration model, wherein the filter kernels are not part of the neural network-based image restoration model.
The target data may include at least some of the input image, the at least some input data of the layers, or at least some output data of the plurality of layers.
The determining the auxiliary data may include determining first auxiliary data by filtering the target data with a first filter kernel of the plurality of filter kernels, and determining second auxiliary data by filtering the target data with a second filter kernel of the plurality of filter kernels.
The determining the new input data may include determining new first input data by combining the first auxiliary data with first input data of a first layer of the layers of the neural network-based image restoration model, and determining new second input data by combining the second auxiliary data with second input data of a second layer of the layers of the neural network-based image restoration model.
The first filter kernel and the second filter kernel may correspond to a Gaussian filter, the second layer may be closer to an output layer of the neural network-based image restoration model than the first layer, and a size of the second filter kernel may be greater than a size of the first filter kernel.
The determining the new input data may include determining new first input data by combining the first auxiliary data and the second auxiliary data with first input data of a first layer of the layers of the neural network-based image restoration model.
The determining the first auxiliary data and the determining the second auxiliary data may be performed in parallel with respect to each other.
The target data may be the input image, the determining the auxiliary data may include selecting the plurality of filter kernels according to respective having different filter parameters thereof, and determining the auxiliary data based on filtering results corresponding to the plurality of filter kernels by filtering the input image in parallel by the plurality of filter kernels, and the generating the restored image may include generating, based on the new input data according to the auxiliary data, the restored image by sequentially executing the plurality of layers of the neural network-based image restoration model.
The plurality of filter kernels may include a first filter kernel corresponding to a Gaussian filter for a first layer of the layers, the determining the auxiliary data may include generating a first filtering result by filtering the target data with the first filter kernel, and determining first auxiliary data corresponding to a difference between the target data and the first filtering result, and the determining the new input data may include determining new first input data by concatenating first input data of the first layer with the first auxiliary data.
The generating the restored image may include generating the restored image based on a difference between an output image of the neural network-based image restoration model and the input image.
At some of the determining the auxiliary data and the determining the new input data may be performed through image signal processing (ISP) of a sensor that captures the input image, and the rest of the determining the auxiliary data and the determining the new input data may be performed through ISP of an application processor (AP).
In one general aspect, an electronic device includes a camera configured to generate an input image, one or more processors, storage hardware storing instructions configured to, when executed by the one or more processors, cause the one or more processors to: receive the input image, determine auxiliary data corresponding to a plurality of filter kernels by filtering target data with the plurality of filter kernels, determine new input data by combining the auxiliary data with at least some input data of layers of a neural network-based image restoration model, and generate a restored image of the input image by executing the neural network-based image restoration model based on the new input data.
The target data may include at least some of the input image, the at least some input data of the plurality of layers, or at least some output data of the plurality of layers.
The instructions may be further configured to cause the one or more processors, to determine the auxiliary data: determine first auxiliary data by filtering the target data with a first filter kernel of the plurality of filter kernels, and determine second auxiliary data by filtering the target data with a second filter kernel of the plurality of filter kernels.
The instructions may be further configured to cause the one or more processors, to determine the new input data: determine new first input data by integrating the first auxiliary data with first input data of a first layer corresponding to any one of the plurality of layers of the neural network-based image restoration model, and determine new second input data by integrating the second auxiliary data with second input data of a second layer corresponding to another one of the plurality of layers of the neural network-based image restoration model.
The instructions may be further configured to cause the one or more processors, to determine the new input data: determine new first input data by combining the first auxiliary data and the second auxiliary data with first input data of a first layer corresponding to any one of the plurality of layers of the neural network-based image restoration model.
The instructions may be further configured to cause the one or more processors to determine the first auxiliary data and the second auxiliary data in parallel.
In one general aspect, a method includes processing an input image with an image restoration model may further include a convolutional neural network (CNN) that includes a first convolution layer and a second convolution layer, the processing including the first convolution layer generating a first feature map as an output thereof, filtering the input image with a filter kernel that may be not part of the restoration model to generate a first filtered image, wherein the processing the image with the restoration model includes providing both the first feature map and the first filtered image as part of an input volume inputted to the second convolution layer, and generating a restored image corresponding to the input image based on an output of the second convolution layer processing the input volume.
The generating the restored image may include finding a difference between an output of the CNN and the input image, and wherein the output of the CNN may be based on the output of the second convolution layer.
Providing the first filtered image may increase a number of channels that may be input to the second convolution layer.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.
Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same or like elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.
The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known after an understanding of the disclosure of this application may be omitted for increased clarity and conciseness.
The features described herein may be embodied in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.
The terminology used herein is for describing various examples only and is not to be used to limit the disclosure. The articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items. As non-limiting examples, terms “comprise” or “comprises,” “include” or “includes,” and “have” or “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.
Throughout the specification, when a component or element is described as being “connected to,” “coupled to,” or “joined to” another component or element, it may be directly “connected to,” “coupled to,” or “joined to” the other component or element, or there may reasonably be one or more other components or elements intervening therebetween. When a component or element is described as being “directly connected to,” “directly coupled to,” or “directly joined to” another component or element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.
Although terms such as “first,” “second,” and “third”, or A, B, (a), (b), and the like may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms. Each of these terminologies is not used to define an essence, order, or sequence of corresponding members, components, regions, layers, or sections, for example, but used merely to distinguish the corresponding members, components, regions, layers, or sections from other members, components, regions, layers, or sections. Thus, a first member, component, region, layer, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, or section without departing from the teachings of the examples.
Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term “may” herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.
Various filters, which are designed based on typical signal processing theories, may be used to remove a degradation element. The degradation element may be removed by applying a filter to an image (or a feature map therefrom) through an operation such as a convolution operation. Typical signal processing-based filters may provide a software and/or hardware implementation that may be accelerated. However, in real-world applications, the typical signal processing-based filters may have a threshold on performance.
Removing noise by using a deep neural network (DNN) may perform better than using signal-processing filters. For example, a DNN based on a convolutional neural network (CNN) may perform a convolution operation on an input feature of each layer and a kernel thereof to extract an input feature of the next layer, and a CNN model may be configured by stacking convolution layers.
The DNN may perform better than signal processing-based filters, but with greater memory and computation time. A CNN having a cascade structure may not be parallelized, and operation time may dramatically (e.g., exponentially) increase as the number of layers increases. Accordingly, the DNN may not be practical for low-power or real-time applications.
The image processing apparatus 100 may generate the restored image 102 corresponding to the input image 101 by using an image restoration model 110 and a filter kernel 121. The image restoration model 110 may include a neural network model. A neural network model may be a DNN that includes a plurality of layers. The plurality of layers may include an input layer, at least one hidden layer, and an output layer. The hidden layer may be also referred to as an intermediate layer, and the output layer may be also referred to as a final layer.
The DNN may be a fully connected network (FCN), a CNN, and a recurrent neural network (RNN), or other suitable type of neural network, or combinations thereof. For example, at least some of the plurality of layers in the neural network may correspond to a CNN and the others may correspond to an FCN. In this case, the CNN portion of the DNN may be referred to as convolutional layers and the FCN portion of the DNN may be referred to as fully connected layers.
In the CNN, data input to a given layer may be referred to as an input feature map and data output from a given layer may be referred to as an output feature map (in practice, there may be multiple channels/feature maps exchanged between two layers). An output feature map of one layer may be the input feature map of the next layer. The input feature maps and output feature maps may be referred to as feature representations or activation data. The input feature map of the input layer may be an input image.
The neural network may be trained based on deep learning to perform inference suitable for a training purpose by mapping input data and output data that are in a nonlinear relationship to each other. The deep learning may be a machine learning technique for applications such as image or speech recognition, from a big data set. The deep learning may be a process of solving an optimization issue to find a point at which energy is minimized while training a neural network by using prepared training data.
Through supervised or unsupervised deep learning, weights corresponding to an architecture or model of the neural network may be obtained. Through the weights, the input data and the output data may be mapped to each other. For example, when the width and depth of the neural network are sufficiently large, the neural network may have sufficient capacity to implement an arbitrary function. When the neural network is trained on a sufficiently large quantity of training data through an appropriate training process, an optimal performance may be achieved.
The neural network may be expressed as being trained in advance, in which “in advance” means “before” the neural network is started. The “started” neural network may indicate that the neural network may be ready for inference. For example, “start” of the neural network may include loading of the neural network in a memory, or an input of input data for inference to the neural network after the neural network is loaded in a memory.
The filter kernel 121 may be a typical signal processing-based filter (e.g., an image-processing filter). For example, the filter kernel 121 may be a Gaussian filter, a median filter, a Sobel filter, and/or the like. However, the foregoing examples are merely examples, and the filter kernel 121 may be other types of filters. There may be multiple filter kernels 121. The filter kernel(s) 121 may not be kernels of neural network layers, e.g., the filter kernel(s) may not be used as CNN kernels. That is, the filter kernels are to be distinguished from kernels of any convolution layers of a neural network. For example, the filter kernels may not include trained/learned weights (or at least values of the filter kernels may not be learned within the context of the image processing apparatus 100 or the images that it is restoring). In other words, the filter kernel(s) 121 may be kernels typically used in digital signal processing of images and may be convolved over padded input data so that a filter kernel's output data has a same dimension as its input data.
The filter kernels 121 may be identified through respective filter parameter. For example, a filter parameter may include the type and characteristics of an associated filter. The identified characteristics of the filter may include a filtering factor, the size of the filter, and/or the like. For example, the filter type may be a Gaussian filter, the filtering factor may be 6 and p, and the size of the filter may be w*h. The a factor may be a standard deviation, the p factor may be an average, w may be a width, and h may be a height. As noted, the image processing apparatus 100 may use a plurality of filter kernels, including the filter kernel 121, for image restoration. The number of filter kernels may be the same as the number of layers of the image restoration model 110. Alternatively, the number of filter kernels may be greater or less than the number of layers of the image restoration model 110.
The image processing apparatus 100 may substantially reduce an operation quantity and memory needed for image restoration by decreasing the number of layers of the image restoration model 110 and by supplementing the image restoration model 110 with the filter kernel 121 that is based on signal processing. The image processing apparatus 100, despite using a DNN with less layers than a typical DNN (which is implemented without a limit on the number of layers) may provide better or at least similar image restoration performance than a typical DNN. In addition, the DNN with less layers may enable faster image restoration with less memory than the typical DNN. Accordingly, the image processing apparatus 100 may be applied to low-power or real-time applications, for example.
More specifically, the image processing apparatus may perform an extracting operation 221, based on the first target data 2211 and the first filter kernel 2212. The first target data 2211 may include at least some of the input image 201 and the first input data 2111. The extracting operation 221 may include a filtering operation by using the first filter kernel 2212. For example, the filtering operation may include convolving the first filter kernel 2212 over first target data 2211. The image processing apparatus may filter the first target data 2211 with the first filter kernel 2212 and generate a first filtering result. If a second (or more) extracting operation 222 is included, the second extracting operation 222 may be performed in a like manner.
The first auxiliary data 2213 may be the first filtering result or a result of an additional operation between the first target data 2211 and the first filtering result. For example, the first auxiliary data 2213 may be a difference between the first target data 2211 and the first filtering result. For example, when the first filter kernel 2212 is a low pass filter (LPF) (e.g., a Gaussian LPF), the first filtering result may correspond to a low-frequency element of the first target data 2211 and the additional operation result (the first auxiliary data 2213) may correspond to a high-frequency element of the first target data 2211.
The image processing apparatus may fuse (e.g., combine) the first auxiliary data 2213 with the first input data 2111 to determine the new first input data 2112. For example, fusion may be performed through concatenation or other methods of combining (concatenation may involve adding the auxiliary data as an additional feature map or image to be inputted to a network layer of an image, e.g., as an additional part of an input volume). The first auxiliary data 2213 may be merged into an additional channel of the first input data 2111 through concatenation in a channel dimension and may serve as a basis for determining the new first input data 2112. Without the fusion, the same as the first input data 2111 serves as the new first input data 2112 (i.e., the first input data 2111 is input to the first layer 211).
The image processing apparatus may execute the first layer 211 to generate the second input data 2121 as an output of the first layer 211. Executing the image restoration model 210 or the first, second, and/or nth layers 211, 212, and 213 may be involve processing a network operation related to the image restoration model 210 or the first, second, and/or nth layers 211, 212, and 213. For example, any layer of the first, second, and/or nth layers 211, 212, and 213 may be a convolutional layer, and the image processing apparatus may perform a convolution operation between the layer and a weight kernel of the layer and determine an output of the layer. Alternatively, when the layer further requires an activation operation (e.g., a rectified linear unit (ReLu)) in addition to the convolution operation, the image processing apparatus may perform the activation operation on a convolution operation result to generate an output of the layer.
The image processing apparatus may process the second layer 212 responsive to an operation on the first layer 211. The image processing apparatus may determine the second auxiliary data 2223 through an extracting operation 222 based on the second target data 2221 and the second filter kernel 2222 and determine new second input data 2122 by fusing the second auxiliary data 2223 with the second input data 2121. For example, the extracting operation 222 may use input data of the first layer 211 and/or output data of the first layer 211 as the second target data 2221. In this case, the input data of the first layer 211 may be the new first input data 2112 and the output data of the first layer 211 may be the second input data 2121. The image processing apparatus may execute the second layer 212 with the new second input data 2122. The extracting and fusing of the first and second auxiliary data 2213 and 2223 may be further performed on several subsequent layers, such as the nth layer 213, and accordingly, the output image 202 may be determined.
When the first and second target data 2211 and 2221 correspond to the input image 201, the first and second auxiliary data 2213 and 2223 may be generated in parallel. In this case, a hardware configuration for generating the first and second auxiliary data 2213 and 2223 in parallel may be provided. The hardware configuration may serve as an accelerator for generating the first and second auxiliary data 2213 and 2223 in parallel, and accordingly, an image restoration time may decrease. A structure of deriving auxiliary data by using the first and second filter kernels 2212 and 2222 may not correspond to a DNN's cascade structure. Therefore, when the input image 201 is the first and second target data 2211 and 2221, a process of generating the first and second auxiliary data 2213 and 2223 may be parallelized (the input image 201 being available at the onset). For example, the first and second auxiliary data 2213 and 2223 may be completely prepared before or at the beginning of executing the image restoration model 210. Because the generation of the first and second auxiliary data 2213 and 2223 may not delay the execution of the image restoration model 210, a time of deriving the output image 202 may decrease.
In an example, the auxiliary data 300 may include a same number of the auxiliary datum as the number of layers in the image restoration model 310. For example the auxiliary data 300 may include first, second, third, and fourth auxiliary data 301, 302, 303, and 304 and the image restoration model 310 may include initial, intermediate, intermediate, and final layers 311, 312, 313, and 314. In such example, the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may respectively fuse (e.g. join) with the initial, intermediate, intermediate, and final layers 311, 312, 313, and 314. An intermediate layer may correspond to a hidden layer, and a final layer may correspond to an output layer.
In another example, two or more of the first, second, third, and fourth auxiliary data 301, 302, 303, and 304, that is, a plurality of pieces of auxiliary data, may fuse with at least one input of any layer of the image restoration model 310. For example, the first auxiliary data 301 and the second auxiliary data 302 may fuse (combine or join) with an input of the initial layer 311, and the third auxiliary data 303 and the fourth auxiliary data 304 may fuse with an input of the final layer 314. In another example, the first, second, third, and fourth auxiliary data 301, 302, 303, and 304 may fuse with an input of the initial layer 311. There may be various other examples of a fusion point, other than the foregoing examples. A fusion pattern which is suitable for a current restoration task (e.g., noise removal, blur removal, image enhancement, etc.), may be adopted for a restoration process of the current restoration task. In other words, a fusion pattern may be any combination(s) of pieces of auxiliary data with fused with any input(s) of restoration model layers.
The image processing apparatus may determine new first input data by fusing the first auxiliary data with first input data of a first layer 411. When the first layer 411 is an initial layer, the first input data of the first layer 411 may be the input image 401. The image processing apparatus may concatenate the first auxiliary data with the first input data in a channel dimension. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1+1. The image processing apparatus may execute the first layer 411 by inputting the new first input data into the first layer 411. For example, the first layer 411 may be a convolutional layer.
The image processing apparatus may determine second auxiliary data by using the input image 401 and a second filter kernel of the filter kernels 420 (to generate second auxiliary data) and determine new second input data by fusing the second auxiliary data with second input data of a second layer 412; the new second input data is input to the second layer 412. The image processing apparatus may execute a third layer 413, based on third auxiliary data, which is determined through an extracting operation 433 and may additionally perform the foregoing operation on other filter kernels and other layers. The second layer 412 and the third layer 413 may each include a respective convolutional layer and an activation layer (as used herein “activation layer” may refer to an activation function). A final layer of an image restoration model 410 may also be a convolutional layer. Accordingly, an output image 402 may correspond to a degradation element (e.g., a high-frequency element) of the input image 401. The image processing apparatus may generate a restored image of the input image 401 by subtracting the output image 402 from the input image 401.
Different filter parameters may be stored and respectively associated with the filter kernels 420 that they represent, which may facilitate selection of different filter kernels for different tasks. For example, a filter parameter may include the type and characteristics of a corresponding filter. The characteristics of the filter may include a filtering factor, the size of the filter, and/or the like. The filter parameters of filter kernels may be set based on the operational characteristics of the first, second, and third layers 411, 412, and 413. For example, when each of the first, second, and third layers 411, 412, and 413 performs a convolution operation, a receptive field may gradually increase toward a following layer of the first, second, and third layers 411, 412, and 413. Accordingly, the filter parameter may be set such that a filter size may increase toward a following layer of the first, second, and third layers 411, 412, and 413. For example, when defining that the filter size (filter parameter) of the first filter kernel is k1, the filter size of the second filter kernel is k2, and the filter size of the third filter kernel is k3, the filter size may be k1<k2<k3. In this case, the filter size of filter parameters of the filter kernels 420 may affect another element (e.g., a filtering factor). For example, when the filter kernels 420 corresponds to a Gaussian filter, the filter size may be set to increase toward a following layer, and accordingly, a filtering factor (e.g., a) of the filter kernels 420 may be changed (e.g., an increase of a). As a result, a receptive field may gradually increase toward a following layer.
Although the examples of
When the input image 401 is used as filter kernel target data, extracting operations 431, 432, and 433 may be parallelized. The image processing apparatus may perform the extracting operations 431, 432, and 433 in parallel and prepare auxiliary data for the first, second, and third layers 411, 412, and 413 and may use the prepared auxiliary data according to an execution sequence of the first, second, and third layers 411, 412, and 413. For example, when executing the third layer 413, the image processing apparatus may use prepared third auxiliary data to determine new third input data of the third layer 413. The term “prepared” means that auxiliary data (e.g., the third auxiliary data) of a current layer (e.g., the third layer 413) is prepared while (or before) the previous layer (e.g., the second layer 412) is being executed. Accordingly, the auxiliary data of the current layer may be readily available when an operation result of the previous layer is output, and thus, a time delay due to the generation of the auxiliary data may not occur.
In another example, the image processing apparatus may determine the second filtering result by (i) filtering the output data 4112 of the first layer 411 with the second filter kernel or by (ii) filtering the input data 4111 and the output data 4112 of the first layer 411 with the second filter kernel. The extracting operations 436 and 437 may respectively use input data and/or output data of the second layer 412 and the third layer 413 as their respective target data.
When input data and/or output data are used as the target data, a sub-neural network may be used for the extracting operations 435, 436, and 437. For example, the sub-neural network of an extracting operation may include a convolutional layer (e.g., a 1*1 convolutional layer). The sub-neural network may decrease a channel dimension of the input data and/or the output data when it is used as target data for a filtering/extraction operation, and in the decreased channel dimension, a filtering/extraction operation by using the filter kernels 420 may be performed. The sub-neural network may decrease an operation quantity of the extracting operations 435, 436, and 437 and related subsequent operations.
The image processing apparatus may determine new first input data by fusing the first auxiliary data and the second auxiliary data with first input data of a first layer 511. When the first layer 511 is an initial layer of the image restoration model 510, the first input data of the first layer 511 may be the input image 501. For example, the image processing apparatus may concatenate the first auxiliary data and the second auxiliary data with the first input data in a channel dimension. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1 plus the number of channels of the first input data. For example, if the auxiliary data is black-and-white image data, then the number of channels of the new first input data may be c1+2 (1 channel for each the outputs of the extracting operations 531 and 532). If the auxiliar data is 3-channel color data, the number of channels of the new first input data may be c1+6 (3 channels for each of the outputs of the extracting operations 531 and 532). However, c1+2 and c1+6 are merely examples, and the number of channels is not limited thereto. The image processing apparatus may execute the first layer 511 by inputting the new first input data into the first layer 511. For example, the first layer 511 may be a convolutional layer (for example, when the image restoration model 501 includes a CNN).
The image processing apparatus may determine third auxiliary data by using the input image 501 and a third filter kernel of the filter kernels 520, may determine fourth auxiliary data by using the input image 501 and a fourth filter kernel of the filter kernels 520, may determine new nth input data by fusing the third auxiliary data and the fourth auxiliary data with nth input data of an nth layer 513, and may input the new nth input data to the nth layer 513. An n−1th layer 512 may include a convolutional layer and an activation layer, and the nth layer 513 may correspond to a convolutional layer. The image processing apparatus may generate a restored image of the input image 501 by subtracting an output image 502 from the input image 501 (as used herein “subtract” is operand-order agnostic; “subtracting” any two operands also means finding a difference between the two operands).
The filter kernels 520 may have different filter parameters (records of information about the filter kernels 520, stored, e.g., in memory and available before/during image restoration). For example, a filter parameter may include the type and characteristics of a correspondingly associated filter. The characteristics of the filter stored in the filter parameter (filter record) may include information about the associated filter kernel, for example a filtering factor, the size of the filter, and/or the like. The filter parameter(s) may be set based on the operational characteristics of the layers 511, 512, and 513. For example, a receptive field may gradually increase toward a following layer of the layers 511, 512, and 513 and the filter parameter may be set such that a filter size increases toward the following layer of the layers 511, 512, and 513. For example, when the filter sizes of first, second, third, and fourth filter kernels are respectively k1, k2, k3, and k4, the filter sizes may be k1=k2<k3=k4. In this case, a change in the filter size may affect an element (e.g., a filtering factor) other than the filter size among the filter parameters of the filter kernels 520.
Since, in the example of
The image processing apparatus may determine new first input data by fusing the first auxiliary data, the second auxiliary data, and the third auxiliary data with first input data of a first layer 611. When the first layer 611 is an initial layer, first input data of the first layer 611 may be the input image 601. For example, the image processing apparatus may concatenate the first auxiliary data, the second auxiliary data, and the third auxiliary data with the first input data in a channel direction. When the number of channels of the first input data is c1, the number of channels of the new first input data may be c1+3 or c1+9. c1+3 may represent black-and-white data and c1+9 may represent color data. However, c1+3 and c1+9 may just be examples, and the number of channels is not limited thereto. The image processing apparatus may execute the first layer 611 by inputting the new first input data into the first layer 611. For example, the first layer 611 may correspond to a convolutional layer. A second layer 612 and a third layer 613 may each include a respective convolutional layer and a respective activation layer (activation function). When an output image 602 is generated through the execution of the first, second, and third layers 611, 612, and 613, the image processing apparatus may generate a restored image of the input image 601 by subtracting the output image 602 from the input image 601. Although pieces of auxiliary data based on three extracting operations 631, 632, and 633 fuse with input data of the first layer 611 in the examples described with reference to
For example, the sensor-ISP 721 may generate a Bayer image 706, based on sensor data according to a Bayer pattern 701, and the AP-ISP 722 may generate a red, green, and blue (RGB) image 707 based on the Bayer image 706. Although
The image restoration model 730 may improve the quality of the RGB image 707 by improving the performance of the restoration tasks, such as denoising, demosaicing, white balancing, and enhancement. Combining the image restoration model 730 with the filter kernels 740 may allow a decreased number of layers of the image restoration model 730, may increase the speed of image restoration, and may decrease memory use. In addition, load balancing/sharing using the sensor-ISP 721 and the AP-ISP 722 may optimize the restoration tasks. Accordingly, the image processing apparatus 100 may be applied to low-power or real-time applications.
The processor 1210 may execute instructions and functions to be executed by the electronic device 1200. For example, the processor 1210 may process instructions stored in the memory 1220 or the storage device 1240. The processor 1210 may perform one or more of the operations described above with reference to
The camera 1230 may capture a photo and/or a video. A capturing result may correspond to an input image such as various input images discussed above. The storage device 1240 may include a computer-readable storage medium or a computer-readable storage device (which are not signals per se). The storage device 1240 may store a greater storage capacity than the memory 1220 and may store data in a non-volatile way. For example, the storage device 1240 may include a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or other non-volatile memories.
The input device 1250 may receive an input from a user through a keyboard and a mouse, through a touch input, a voice input, an image input, or the like. For example, the input device 1250 may include a keyboard, a mouse, a touch screen, a microphone, or any other device for detecting the input from the user and transmitting the detected input to the electronic device 1200. The output device 1260 may render an output of the electronic device 1200 to the user through a visual, auditory, or haptic channel. The output device 1260 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. The network interface 1270 may communicate with an external device through a wired or wireless network.
The target data may include at least some of the input image, at least some of the input data of the plurality of layers, and at least some of output data of the plurality of layers. For example, the target data may be the input image, input data of a layer corresponding to a filter kernel, or input data of the previous layer of the layer corresponding to the filter kernel. The input data of the layer corresponding to the filter kernel may correspond to output data of the previous layer.
Operation 1320 may include determining first auxiliary data by filtering the target data with a first filter kernel of the plurality of filter kernels and determining second auxiliary data by filtering the target data with a second filter kernel of the plurality of filter kernels.
Operation 1330 may include determining new first input data by fusing the first auxiliary data with first input data of a first layer and determining new second input data by fusing the second auxiliary data with second input data of a second layer, in which the first layer corresponds to any one of the plurality of layers of the neural network-based image restoration model and the second layer corresponds to another one of the plurality of layers of the neural network-based image restoration model.
The first filter kernel and the second filter kernel may correspond to a Gaussian filter, the second layer may be closer to the output layer of the neural network-based image restoration model than the first layer, and the size of the second filter kernel may be greater than the size of the first filter kernel.
Operation 1330 may include determining the new first input data by fusing the first auxiliary data and the second auxiliary data with the first input data of the first layer corresponding to any one of the plurality of layers of the neural network-based image restoration model. For example, the first layer may correspond to an initial layer, an intermediate layer, or a final layer of the neural network-based image restoration model.
Determining the first auxiliary data and determining the second auxiliary data may be performed in parallel.
The target data may correspond to (or may be) the input image. Operation 1320 may include defining the plurality of filter kernels having different filter parameters from one another and determining auxiliary data, based on filtering results corresponding to the plurality of filter kernels by filtering the input image with the plurality of filter kernels in parallel. Operation 1340 may include generating, based on new input data according to the auxiliary data, a restored image by sequentially executing the plurality of layers of the neural network-based image restoration model.
The plurality of filter kernels may include the first filter kernel corresponding to a Gaussian filter for the first layer of the plurality of layers. Operation 1320 may include generating a first filtering result by filtering the target data with the first filter kernel and determining the first auxiliary data corresponding to a difference between the target data and the first filtering result. Operation 1330 may include determining the new first input data by concatenating the first auxiliary data with the first input data of the first layer.
Operation 1340 may generate the restored image, based on a difference between an output image of the neural network-based image restoration model and the input image.
Determining the auxiliary data or determining the new input data may be performed by a sensor-ISP, whichever of the two is not performed by the sensor-ISP may instead be performed by an AP-ISP. For example, operations related to certain layers of the neural network-based image restoration model may be performed by the sensor-ISP and operations related to the other layers of the neural network-based image restoration model may be performed by the AP-ISP. In other words, in some embodiments, some of the determining of the auxiliary data and the determining the new input data is performed by a sensor-ISP, and the rest of the determining of the auxiliary data and the determining the new input data is performed by the AP-ISP, and the division of computation between the AP-ISP and sensor-ISP can vary.
In addition, the descriptions provided with reference to
The computing apparatuses, the vehicles, the electronic devices, the processors, the memories, the image sensors, the displays, the information output system and hardware, the storage devices, and other apparatuses, devices, units, modules, and components described herein with respect to
The methods illustrated in
Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.
The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.
While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.
Therefore, in addition to the above disclosure, the scope of the disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2022-0087068 | Jul 2022 | KR | national |
