This application is a U.S. National Phase Application under 35 USC 371 of International Application PCT/JP2005/011004 filed Jun. 9, 2005.
The present invention relates to an imaging system adapted to make use of an image input means having a reduced number of pixels to generate a high-resolution image.
Various methods that make use of image data having a reduced number of pixels to generate high-resolution images have been proposed for use with imaging systems such as video cameras. As set forth typically in JP(A)10-69537, there is a method wherein an ultra-resolution technique is used with a low-resolution image comprising multiple frames having displacements to generate a high-resolution image. Ultra-resolution processing is a technique where two or more images having displacements at the sub-pixel level are taken, and they are then combined together into one single high-definition image after factors responsible for their deteriorations are canceled out.
By the way, there is an imaging system wherein, as is the case with a video camera, there is some limitation to the number of clocks per frame, and an imaging device has more pixels than an output image has. To implement effective ultra-resolution processing when such an image system is used, it is required that data be read out of only a part of the imaging device, and ultra-resolution processing be applied to that area alone. However, a change in the angle of view for implementing ultra-resolution then requires a change in the optical image magnification.
When the technique set forth in Patent Publication 1 is applied to the generation of a high-resolution image using image data having a reduced number of pixels, there is a problem that processing becomes complicated because, as mentioned just above, the change in the angle of view for implementing ultra-resolution processing requires a change in the optical image magnification.
In view of the above problems, an object of the present invention is to provide an imaging system that enables the size of the area to be imaged to be electronically changed without causing variations in the number of clocks per frame, and ultra-resolution processing to be applied to the area to be imaged.
(1) According to the invention, the aforesaid object is achieved by the provision of an imaging system for electronically obtaining an image of a subject, characterized by comprising an optical image-formation means adapted to form the image of the subject on an imaging device, an imaging device capable of producing an image signal of a given area, an area setting portion adapted to set an output area from said imaging device, a means adapted to select a read rule for said imaging system depending on the size of an area set at said area setting portion, and a means adapted to generate a high-resolution image from image signals of multiple frames produced out of said imaging device.
The invention (1) is equivalent to an embodiment shown in
(2) The aforesaid invention (1) is further characterized in that the read rule for said imaging device is such that irrespective of the size of said output area, the total number of clocks upon reading of pixels is constant. The invention (2) is equivalent to an embodiment of
(3) The aforesaid invention (2) is also characterized by further comprising a means adapted to make said read rule for the imaging device different for each frame. The invention (3) is equivalent to the embodiment of
(4) Further, the aforesaid invention (3) is further characterized in that the means adapted to generate a high-resolution image from said image signals of multiple frames comprises a means adapted to estimate a motion between multiple frames, a means adapted to use image signals of the multiple frames of which the motion is estimated to estimate a high-resolution image signal, and a means by which a mutual identical read rule is selected when said motion between multiple frames is estimated.
The invention (4) is equivalent to an embodiment of
(5) The aforesaid invention (4) is further characterized by comprising a means by which frames in compliance with the same read rule are selected for motion estimation when said motion estimation between multiple frames is implemented, wherein the means implements computation for estimation of a motion between continuous frames. The invention (5) is equivalent to an embodiment shown in
(6) The aforesaid invention (2) or (3) is further characterized in that the read rule for said imaging device is a cull read adapted to read pixels discretely. The invention (6) is equivalent to the embodiment of
(7) The aforesaid invention (6) is further characterized by comprising a means adapted to correct distortion due to cull read after cull read from said imaging device. The invention (7) is equivalent to an embodiment of
(8) The aforesaid invention (7) is further characterized in that said distortion correction processing is pixel computation processing within the same frame. The invention (8) is equivalent to an embodiment of
According to the imaging system of the invention, the size of the area to be imaged can be electronically changed with no fluctuation of the number of clocks per frame, and ultra-resolution processing can be applied to the area captured.
Some embodiments of the invention are now explained with reference to the accompanying drawings.
The read image signals are stored in n image memories 105-1 to 105-n, where n is the number of images needed for ultra-resolution processing. The ultra-resolution processing comprises a motion estimation block 107 and a high-resolution image estimation block 108 adapted to estimate image data having a high-resolution pixel sequence. A selector 106 selects a basic reference for motion estimation and an image that is estimated in terms of motion.
Referring to pixels in the row direction with respect to ODD and EVEN, a sequence of RGRG . . . at the first row, a sequence of GBGB . . . at the second row, a sequence of RGRG . . . at the third row, a sequence of GBGB . . . at the fourth row and the like appear repeatedly. Referring to pixels in the column direction, a sequence of RGRG . . . at the first column, a sequence of GBGB . . . at the second column, a sequence of RGRG . . . at the third column, a sequence of GBGB . . . at the fourth row and the like appear repeatedly.
In
At S6, the discrete similarity map prepared at S5 is complemented thereby searching and finding the extreme value for the similarity map. A transformation motion having that extreme value defines an estimation motion. For the purpose of searching the extreme value for the similarity map, there is parabola fitting, spline interpolation or the like. At S7, whether or not motion estimation has been made of all reference images of interest is determined. At S8, if not, the frame number of the reference image is increased by 1 to resume the processing of S3, keeping on the read processing of the next image. When motion estimation has been made of all reference images of interest, the processing comes to an end.
Here, y is a low-resolution image, z is a high-resolution image, and A is an image transformation matrix indicative of an imaging system including an inter-image motion, PSF, etc.; g(z) includes a restraint term or the like, in which care is taken of image smoothness and color correlation; and λ is a weight coefficient. For the minimization of the estimation function, for instance, the steepest descent method is used. At S16, when f(z) found at S15 is already minimized, the processing comes to an end, giving the high-resolution image z. At S17, when f(z) is not yet minimized, the high-resolution image z is updated to resume the processing at S15.
First, of image data as many as multiple frames recorded in the image memories 101-1 and 105-n, any one image that defines a basis is given to the interpolation and enlargement block 61 where the image is interpolated and enlarged. The interpolation and enlargement method used here, for instance, includes bilinear interpolation and bicubic interpolation. The interpolated and enlarged image is given to the convolution integration block 62, and subjected to convolution integration along with PSF data sent from the PSF data holding block 63. And of course, the motion of each frame is here taken into the image data. The interpolated and enlarged image data are at the same time sent to the image buildup block 69 for accumulation there.
Image data to which convolution computation is applied are sent to the image comparison block 64 where, on the basis of the motion of each frame found at the motion estimation block 107, they are compared at a proper coordinate position with taken images given out of the imaging block. The difference compared at the image comparison block 64 is forwarded to the multiplication block 65 for multiplication by the value per pixel of the PSF data given out of the PSF data holding block 63. The results of this computation are sent to the superposition addition block 66, where they are disposed at the corresponding coordinate positions. Referring here to the image data from the multiplication block 65, the coordinate positions displace little by little with overlaps, and so those overlaps are added on. As the superposition addition of one taken image of data comes to an end, the image data are forwarded to the accumulation addition block 67.
At the accumulation addition block 67, successively forwarded data are built up until the processing of data as many as frames gets done, and one each frame of image data are added on following the estimated motion. The added image data are forwarded to the update image generation block 68. At the same time, the image data built up at the image accumulation block 69 are given to the update image generation block 68, and two such image data are added with a weight to generate update image data.
The generated update image data are given to the iterative computation determination block 610 to judge whether or not the computation is to be repeated on the basis of the iterative determination value given out of the iterative determination value holding block 611. When the computation is repeated, the data are forwarded to the convolution integration block 62 to repeat the aforesaid series of processing, and when not, the generated image data are outputted. The motion for each frame is given from the motion estimation block 107 to the PSF data held at the aforesaid data holding block 63, because computation at a proper coordinate position becomes necessary for convolution integration.
The motion estimation for the image that is subjected to cull read as shown in
With methods (1) and (2), reading is implemented in compliance with the same read rule at ODD and EVEN, respectively, as shown in
The estimation of the values of missing pixels here could be made by either such two-stage processing of interpolation and reduction as shown in
Formula (2) given below represents in a matrix form a method of filling up the missing pixels by virtue of linear interpolation and using linear interpolation as a size change.
In the operation at the second term on the right side of formula (2), image data sampled like R(i), G(i+1), R(i+2), R(i+4), G(i+5), G(i+7), R(i+8) . . . are interpolated to generate R(i), G(i+1), R(i+2), G−(I+3), R(i+4), G(i+5), R−(i+8), G(i+7) . . . , and in the operation at the first term on the right side, transformation of 8 pixels into 6 pixels is implemented by linear interpolation.
These operations are combined into such linear transformation as represented by formula (3), given just below.
The logic table 1 shows the state of one pixel clock in the row direction, wherein there is a pixel data shift like C1<=C2<=C3.
For the estimation of the luminance level of missing pixels, use could be made of not only such primary interpolation at the same channel as described above, cubic interpolation at the same channel and linear interpolation like a sinc function but also an interpolation method using correlations between R, G and B channels.
The value of estimation of the luminance level of missing pixels is given by formula (4) or (5), mentioned just below.
After such discrete reading as shown in
The invention as described above can provide an imaging system in which the size of the area to be imaged can be electronically changed with no fluctuation of the number of clocks per frame, and ultra-resolution processing can be applied to the area captured.
Number | Date | Country | Kind |
---|---|---|---|
2004-172094 | Jun 2004 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2005/011004 | 6/9/2005 | WO | 00 | 1/4/2007 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2005/122554 | 12/22/2005 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
5657402 | Bender et al. | Aug 1997 | A |
6285804 | Crinon et al. | Sep 2001 | B1 |
6330344 | Kondo et al. | Dec 2001 | B1 |
6618081 | Harada et al. | Sep 2003 | B1 |
6750903 | Miyatake et al. | Jun 2004 | B1 |
6906751 | Norita et al. | Jun 2005 | B1 |
7352919 | Zhou et al. | Apr 2008 | B2 |
20030227552 | Watanabe | Dec 2003 | A1 |
20050141047 | Watanabe | Jun 2005 | A1 |
Number | Date | Country |
---|---|---|
4-172778 | Jun 1992 | JP |
4-196775 | Jul 1992 | JP |
7-131692 | May 1995 | JP |
2000-41186 | Feb 2000 | JP |
2002-112096 | Apr 2002 | JP |
2002-369083 | Dec 2002 | JP |
2003-338988 | Nov 2003 | JP |
Number | Date | Country | |
---|---|---|---|
20070268388 A1 | Nov 2007 | US |