Patent Application
20240144449
The aspect of the embodiments relates to a technology for adding a blurring effect or a deblurring effect to an image.
Hitherto, a technology for estimating a point spread function (PSF) that represents the blur or bokeh characteristics of a subject included in an image and adding a blurring effect or a deblurring effect to the image on the basis of the estimated PSF has been known. For example, Japanese Patent Laid-Open No. 2007-306548 discloses a method for reducing (deblurring) image blur included in a captured image by specifying a specific region in the captured image, calculating a PSF with reference to the image of the specific region, and deconvolving the obtained PSF.
One feature of PSF estimation is that some subject patterns to be referenced may degrade PSF estimation results. For example, a fine lattice pattern such as the string portion of a tennis racket appears as a shape in estimated characteristics. That is, it means that the characteristics of subject blur are not accurately estimated. In the method described in Japanese Patent Laid-Open No. 2007-306548, using such an incorrect PSF to perform correction processing may result in degradation of image quality.
An apparatus includes a determination unit configured to determine a specific region in an image, and a point spread function estimation unit configured to perform point spread function estimation for the specific region. The point spread function estimation unit is capable of performing at least one of edge extraction and mask processing on the image, and is configured to estimate a point spread function for the processed image.
Further features of the disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
In the following, an embodiment of the present disclosure will be described in detail with reference to the drawings. An embodiment to be described below is an image capturing apparatus. As an example of the image capturing apparatus, an example in which the present disclosure is applied to a digital camera is described. Note that the present disclosure relates to correction processing to be applied to captured images and more specifically relates to a method in which an image signal is corrected in order to be used in estimation, a PSF is estimated, and a blurring effect or a deblurring effect is added to a specific region of the image on the basis of the estimated PSF. Thus, configurations for image capturing and recording are not essential in the image capturing apparatus. Moreover, in the present embodiment, “bokeh” includes out-of-focus blur due to image defocusing, and “blur” includes camera shake and subject motion blur, for example; however, “bokeh” and “blur” are not limited thereto.
A controller 101 is, for example, a central processing unit (CPU). The controller 101 controls operations of individual blocks of a digital camera 100 by reading out operation programs for the individual blocks of the digital camera 100 from a read-only memory (ROM) 102, loading the operation programs into a random-access memory (RAM) 103, and executing the loaded operation programs.
The ROM 102 is a rewritable non-volatile memory and stores, for example, parameters necessary for operating the individual blocks in addition to the operation programs for the individual blocks of the digital camera 100. The RAM 103 is a rewritable volatile memory and is used as a temporary storage area for data output in the operation of each block of the digital camera 100.
An optical system 104 forms a subject image on an image capturing unit 105. The optical system 104 includes, for example, a fixed lens, a zoom lens that changes the focal length, and a focus lens that adjusts the focus. The optical system 104 also includes an iris and controls the amount of light at the time of image capturing by adjusting an aperture diameter of the optical system using the iris.
The image capturing unit 105 is, for example, an image sensing device such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor. The image capturing unit 105 photoelectrically converts an optical image formed on the image sensing device by the optical system 104 and outputs the obtained analog image signal to an analog-to-digital (A/D) converter 106. The A/D converter 106 performs A/D conversion processing on the analog image signal input thereto, and outputs the obtained digital image data to the RAM 103, so that the RAM 103 stores the digital image data.
An image processing unit 107 performs various types of image processing, such as white balance adjustment, color interpolation, and gamma processing, on the image data stored in the RAM 103 and outputs the resulting image data to the RAM 103. Moreover, the image processing unit 107 performs processing on captured images or on the image data stored in the RAM 103 in order to add a blurring effect or a deblurring effect.
A recording medium 108 is a detachable memory card or the like. In the recording medium 108, images processed by the image processing unit 107 and stored in the RAM 103 and images converted from analog to digital by the A/D converter 106 are recorded as recorded images.
A display 109 is a display device such as a liquid crystal display (LCD). The display 109 displays various types of information in the digital camera 100. For example, the display 109 realizes an electronic viewfinder function by displaying, in a live view manner, a subject image captured by the image capturing unit 105, and plays back and displays an image recorded in the recording medium 108.
An operation input unit 110 includes a user input interface and outputs, to the controller 101, a control signal corresponding to an operation input performed by the user upon detection of the operation input. Examples of the user input interface include a release switch, a setting button, and a mode setting dial. In an embodiment in which the display 109 has a touch panel sensor, the operation input unit 110 also functions as an interface configured to detect a touch operation performed on the display 109.
The configuration and basic operation of the digital camera 100 are described above.
Next, the operation of the image processing unit 107, which is a characteristic of the first embodiment according to the present disclosure, will be described in detail. In the present embodiment, it is intended to capture an image of a subject performing a tennis swing. In general, when capturing images of a subject playing tennis, golf, baseball batting, and other sports that involve swinging a hand-held racket, club, or bat (hereinafter referred to as swing sports), in one embodiment, there is no blur or bokeh on images of the playing subject's face. In contrast, if the swinging regions of the racket and arms are blurred in the swinging direction in an image, the image is more dynamic and conveys the intensity of the play more impressively to the viewer, resulting in a favorable image. In the present embodiment, a method will be described in which a PSF for a swinging region is estimated with high accuracy, and a blurring effect is added on the basis of the estimated PSF.
Blurring effect addition processing performed by the image processing unit 107 will be described with reference to
As illustrated in
Next, in S301 of
In S302, the main subject detector 201 performs an operation for detecting a main subject in the image data 206 captured in S301. In this case, the main subject detector 201 detects the human subject 401. It is sufficient that existing techniques be used as a method for detecting the human subject 401 as the main subject. For example, it is sufficient that a method as disclosed in Japanese Patent Laid-Open No. 2002-77711 be used.
In S303, the PSF estimation region extractor 202 extracts a region including a swinging object (the tennis racket) in the human subject 401 detected by the main subject detector 201. It is sufficient that existing techniques be used as a method for extracting the region of the tennis racket. As in the case of the main subject in S302, the region can be extracted using a method as disclosed in Japanese Patent Laid-Open No. 2002-77711. Alternatively, the user may be caused to specify a PSF estimation region through the operation input unit 110. In the present embodiment, the size of the PSF estimation region is a square region having 256×256 pixels. The extracted region is illustrated in
In S304, the PSF estimation region correction unit 203 applies a filter for extracting edges to the region extracted by the PSF estimation region extractor 202. In this processing step, the string portion of the tennis racket is excluded, which degrades a PSF estimation result. Specifically, high-frequency components included in the PSF estimation region are excluded, and a bandpass filter that enables edge extraction is applied.
An example of characteristics of the bandpass filter is illustrated in
In S305, the PSF estimation unit 204 estimates a PSF using the signal output from the PSF estimation region correction unit 203. Existing techniques can be used to estimate the PSF. As an example, the following is an overview of PSF estimation using an iterative method.
In a case where an image signal of a PSF estimation region is denoted by B, the PSF to be estimated is denoted by K, and an image (latent image) without blur or bokeh in the PSF estimation region is denoted by L, when noise in the image is ignored, the relationship expressed by the following Eq. (1) holds.
B=Conv(K,L) (1)
In Eq. (1), Cony represents a convolution operation. Eq. (1) cannot be solved analytically because both K and L are unknowns. Thus, the following estimation is performed.
First, the latent image L is set to have an appropriate initial value. An image obtained by performing edge extraction processing on a captured image can be used as the initial value of the latent image L.
Subsequently, in an energy function E(K) expressed by the following Eq. (2), where L is held constant and K is an unknown parameter, K that minimizes the energy function E(K) is calculated.
E(K)=argminK(μB−Conv(K,L)∥2+σK2) (2)
In Eq. (2), σ represents the regularization term, which is an L2 norm that takes the sum of squares of each element of the PSF in this case.
Subsequently, in an energy function E(L) expressed by the following Eq. (3), where K is held constant and L is an unknown parameter, L that minimizes the energy function E(L) is calculated using the calculated K.
E(L)=argminL(∥B−Conv(K,L)∥2+σL2) (3)
K that minimizes the energy function E(K) is calculated using the calculated L, and then L that minimizes the energy function E(L) is calculated using the calculated K. This processing is iteratively performed until the calculated K converges to a constant value, so that the PSF is estimated. The description above is the overview of PSF estimation using the iterative method.
In S306, the image correction unit 205 adds a blurring effect to the image data 206 with reference to the signal output by the PSF estimation unit 204. First, a kernel in which blur is emphasized along the direction of the blur is generated with reference to the kernel of
The generated kernel is illustrated in
Note that in a case where, for example, exposure time is used as a reference and where the exposure time is 1/500 second in this embodiment and is changed to about 1/250 second, it is sufficient that the amount (length) and intensity of blur to be emphasized be doubled in the PSF estimation result. A configuration may also be used in which the size of the athletic equipment in the screen or the distance to the subject divided by the focal length at the time of image capturing is used as a reference, and the amount of blur to be emphasized so as to correspond to the value of the reference is predetermined. Alternatively, a configuration may be used in which the levels of emphasis (low, middle, and high) are preset, and one of the levels of emphasis is selected by the user. By appropriately determining the characteristics of blurring effects to be added in this manner, it is possible to generate suitable images.
Although the present embodiment describes a case where a blurring effect is added to the swinging racket region, it is also possible to add a deblurring effect to a region (for example, a face region) where blur (including bokeh) has occurred. In that case, in S303, the PSF estimation region extractor 202 extracts, as a specific region, a face region of the human subject 401 detected by the main subject detector 201. Then, in S304, the PSF estimation region correction unit 203 detects skin color regions from the region extracted by the PSF estimation region extractor 202 so that only the edges of skin color regions of the extracted region become valid, and performs mask processing for skin color regions on the edge extraction results. In this manner, PSF estimation can be prevented from being degraded by the effects of objects with fine patterns that are present around the face, such as a mesh hat, for example. Next, a PSF is estimated using the signal of the corrected PSF estimation region. Lastly, the image correction unit 205 performs an inverse convolution operation between the face region of the image data 206 and the estimated PSF to add a deblurring effect. The inverse convolution operation is expressed by the following Eq. (4).
In Eq. (4), F denotes the Fourier transform, and F−1 denotes the inverse Fourier transform.
In the present embodiment, the size of the PSF estimation region is a square region having 256×256 pixels; however, the size of the PSF estimation region is not limited to this and may be a region other than a square region. For example, in a case where a swinging object is an elongated object like a baseball bat, in one embodiment, the PSF estimation region is also rectangular. When the PSF estimation region has such a shape, it is more likely to prevent another subject present in the background from being included in the PSF estimation region, so that the accuracy of PSF estimation for the swinging object improves. In that case, the corresponding rectangular region is divided into a plurality of squares, a PSF is estimated for each square, and a PSF estimation result can be calculated for the original rectangle by adding up the PSFs in frequency space. Specifically, in a case where the rectangle is divided into n regions, a value (F(K_all)) of the PSF estimation result of the rectangle in frequency space is expressed by the following Eq. (5).
F(K_all)=Σi=0nF(Kn) (5)
Moreover, in a case where a PSF is not estimated for every region obtained as a result of division due to, for example, the load involved in computing, the priority levels of division regions to be used for PSF estimation can be determined with reference to the edge extraction result.
Specifically, the more widely varied the angles of the edges in a region are, the more stable the PSF estimation result becomes. Thus, in one embodiment, such regions are referred to in order to perform estimation. For example, in one embodiment, regions including portions of athletic equipment where the logo of the manufacturer or letters are printed are referred to in order to perform estimation.
In addition, in a case where a circular object is input, edges have less imbalance than in the case of a linear object, and a more stable PSF estimation result can be obtained with respect to the direction of blur. Thus, for example, in the case of a golf club, in one embodiment, the regions including portions of the head of the golf club are used to perform estimation than the region of the shaft of the golf club.
In the following, a second embodiment of the present disclosure will be described. Note that an embodiment to be described below is an image capturing apparatus similarly to as in the first embodiment and will be described using an example in which the present disclosure is applied to a digital camera as an example of an image capturing apparatus.
In the first embodiment, the accuracy of PSF estimation is improved, which enables acquisition of dynamic images. However, in the present embodiment, high-quality dynamic images are generated in a stable manner by additionally performing scene recognition (scene estimation) using machine learning or deep learning techniques.
The configuration of the digital camera according to the second embodiment of the present disclosure is substantially the same as the block diagram in
In
In S1004, the scene recognition unit 907 recognizes (estimates) a scene represented by the image data 906. In this case, the scene recognition unit 907 recognizes the type of sport. It is sufficient that an existing method be used as a method to recognize what kind of sport is being played. For example, it is sufficient that an existing method as disclosed in (Aki SHIBUTANI, Yoshiaki SUGIYAMA, and Yasuo ARIKI, “Sports Kiji no Jidou Hambetsu to Ruiji Shïn no Kensaku”, Information Processing Society of Japan, Proceedings of the 55th National Convention of Information Processing Society of Japan, pp. 65-66 (1997-09)) be used. Subjects playing not only tennis but also sports such as golf or baseball can be recognized.
Furthermore, the scene recognition unit 907 estimates the movement direction of the racket at the time of image capturing from a recognition result of the type of sport, the way in which the digital camera 100 and the human subject 401 face each other, and a position-orientation estimation result of the racket in the image data of the racket region extracted in S1003. Note that, as a method to estimate the position and orientation of the racket, it is sufficient that a method as disclosed in (Ryosuke ARAKI, Kosuke MANO, Takeshi ONISHI, Masanori HIRANO, Tsubasa HIRAKAWA, Takayoshi YAMASHITA, and Hiromitsu FUJIYOSHI, “Gazou Seisei Nettowāku no Gyaku-Dempan ni motozuku Kurikaeshi Koushin ni yoru Buttai Shisei Suitei wo mochiita Buttai Haji”, Annual Conference of The Robotics Society of Japan, 2020) be used.
Processing in S1005 is the same as that in S304 of the first embodiment, and thus description will be omitted.
In S1006, the PSF estimation unit 904 estimates a PSF using the signal output from the PSF estimation region correction unit 903. In PSF estimation according to the present embodiment, an evaluation value of the degree of edge imbalance in the input signal is calculated in addition to the processing in the first embodiment. Specifically, the angle of each edge included in the region is calculated, and the angular variance over the entire region is used as the evaluation value of the degree of edge imbalance. The higher the evaluation value, the more edges of various angles exist in the region. The more edges of various angles there are in the estimation region, the more stable the PSF estimation result in the direction of blur becomes. Thus, the higher the evaluation value of the degree of edge imbalance, the more accurate the PSF estimation result is generated. The PSF estimation unit 904 outputs the PSF estimation result and the evaluation value of the degree of edge imbalance to the image correction unit 905.
In S1007, the image correction unit 905 refers to the signal output from the PSF estimation unit 904 and the output signal from the scene recognition unit 907 to add a blurring effect to the image data 906. The direction (angle) of blur to be added is obtained by combining the direction estimated through PSF estimation and the direction estimated by the scene recognition unit 907 using a weight generated from the evaluation value of the degree of edge imbalance.
As in
DIR_BLUR_F=DIR_BLUR_PSF*α+DIR_BLUR_SCENE(1.0−α) (6)
A kernel in which blur is emphasized along the calculated direction DIR_BLUR_F is generated.
A method for emphasizing blur and a method for adding blur are the same as those in the first embodiment.
With the configuration as in the present embodiment, even in a case where the edges in the region are imbalanced at some angles, the direction in which a swinging object moves can be estimated by also using machine learning techniques or deep learning techniques in, for example, position-orientation estimation. This allows the direction of blur to be calculated with high accuracy even for, for example, sports equipment with many linear portions, such as a baseball bat, and high-quality, dynamic images can be obtained in a stable manner.
Although some embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present disclosure.
While the disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2022-172597, filed Oct. 27, 2022, which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2022-172597 | Oct 2022 | JP | national |
